You cannot watch accuracy in production, because nobody is labelling your camera feeds — you know what the model said, not what was there. So monitoring is entirely proxies, plus a small deliberate sampling routine. The most valuable proxy is free: how often operators dismiss alerts. And the conclusion most teams resist — the majority of apparent drift is physical, not statistical. Retraining on a knocked camera teaches the model that the broken view is normal.
A vision system passes acceptance, goes live, and works. Some months later the operations team trusts it less than they did, nobody can say exactly when that started, and the proposal on the table is to retrain. That proposal is usually premature and occasionally harmful.
The underlying problem is structural. In evaluation you have labelled data, so accuracy is measurable. In production you have camera feeds and no labels, so accuracy is not measurable at all. Everything you can actually watch is a proxy — a signal correlated with quality that needs no ground truth — and knowing which proxy catches which failure is most of the skill.
The signals, and what each one actually catches
Operator behaviour is the best signal and it is free
If the people receiving alerts start closing more of them without acting, precision has fallen. They will know this weeks before any dashboard does, and the data already exists in whatever system shows them the alerts.
Read the other way it is equally useful: a sudden drop in alert volume on a camera that should be busy is far more likely to mean recall collapsed than that the site became safer overnight. And capturing a dismissal reason — even three buttons, "wrong object / already handled / not a real event" — converts a number into a diagnosis at almost no cost.
Most deployments have this data and nobody reads it, which is a strange thing to be true of the only signal that comes from the people who actually know whether the system is working.
Watch the input, not just the output
Simple per-camera statistics over time — mean brightness, contrast, sharpness — move before detection counts do, and they point at a cause rather than a symptom.
A camera knocked during maintenance, a lens gone dirty, a light fitting that failed, a seasonal change in daylight: all of these show up in image statistics early. A detection-rate alert tells you something is wrong. An image-statistics alert usually tells you what, which is the difference between an investigation and a work order.
Most drift is not drift
This is the part teams resist, and it is the most expensive misunderstanding in production vision.
The large majority of apparent degradation on a real site is physical. A camera nudged. Scaffolding up for a month. A failed light. A new product with different packaging. A changed uniform colour. A shelf moved into frame. None of those are distribution shift in any interesting sense — they are the world changing in ways a site visit fixes.
Retraining in response does not solve it. It teaches the model that the degraded condition is normal, and it makes the eventual correction harder, because the model now expects the broken view and will have to be retrained again when someone finally straightens the camera. You have spent a retraining cycle to entrench a fault.
So the first question when a signal moves is always what changed on the site, never what changed in the data. That is the same ordering we argued for in why vision accuracy drops between the demo and the floor: physical fixes first, rule-layer fixes second, model changes last.
What legitimately triggers a retrain
Two cases, and both require labelled evidence rather than a proxy.
Additive change. A new product line, a new class of object, a new uniform or piece of equipment that is now in scope. The model is being asked to recognise something it was never taught. This is unambiguous and it is the easiest case to plan for.
Confirmed decline. A freshly labelled sample shows accuracy genuinely below acceptance, and the physical environment has been checked and ruled out. The second clause is the one that gets skipped.
Everything else — a dip after a maintenance window, one camera misbehaving, a seasonal lighting change that will reverse in two months — is better handled by fixing the cause or simply waiting.
The sampling routine
One thing in the monitoring stack is not a proxy, and it needs to be deliberate: a modest labelled sample drawn at regular intervals from live footage, covering the conditions that matter including the awkward ones.
The value is in comparability rather than volume. A single measurement tells you where you are; a series tells you which way you are going, which is the actual question. Teams consistently find a small recurring sample more useful than a large one-off exercise, and the discipline of keeping to the schedule matters more than the sample size.
Put a date in a calendar and an owner against it. A monitoring plan that depends on someone noticing is not a plan — and quiet degradation is, by definition, the kind nobody notices.
Questions we get asked
Why can't we monitor accuracy directly?
Because accuracy needs the right answer and nobody labels production feeds. You know what the model reported, not what was there. That's why vision systems degrade quietly rather than failing visibly — and why everything useful is a proxy, plus a sampling routine that manufactures a little ground truth on purpose.
What's the most valuable signal?
Operator dismissal and override rate, and it's usually free. More alerts closed without action means precision fell — and the operators know weeks before any dashboard. Read in reverse, a sudden drop in alerts on a busy camera means recall collapsed, not that the site got safer. Capture a dismissal reason and it becomes a diagnosis.
Why watch image statistics?
Because input signals move before output ones and point at the cause. Per-camera brightness, contrast and sharpness over time reveal a knocked camera, dirty lens, failed light or seasonal shift before detection counts change. A detection alert says something is wrong; an image-statistics alert says what.
Is most drift really model drift?
No — most is physical. A nudged camera, scaffolding, a dead light, new packaging, a changed uniform. Retraining on that doesn't fix it, it teaches the model the degraded condition is normal and makes the eventual correction harder. Ask what changed on the site before asking what changed in the data.
What legitimately triggers a retrain?
Additive change — a new product, class, uniform or equipment now in scope. Or a confirmed decline on a freshly labelled sample with the physical environment checked and ruled out. That second clause is the one that gets skipped. Everything else is better fixed at source or waited out.
How often should we sample and label?
On a schedule set in advance and actually kept — the value is comparability, not volume. A modest recurring sample across the conditions that matter beats a large one-off, because one number says where you are and a series says which way you're going. Put an owner and a date against it.
We build on-premise vision systems with the monitoring designed in, because a system nobody is watching degrades silently by definition. If yours is live and trusted less than it was, the dismissal log is the cheapest place to start looking.