An accuracy figure is a measurement of a model against one particular dataset. It is not a property of the model and it is certainly not a property of your site. When a demo number fails to reproduce on a live camera, nothing degraded — the number simply never described your conditions. The gap comes from five mundane places: light, angle, occlusion, class mix and duration. Most of it is recoverable without touching the model, and the fastest fixes usually involve a ladder rather than a GPU.
Every vision project reaches the same awkward meeting. A model that scored impressively in evaluation is now running on real cameras and the operations team is unimpressed. Someone asks what went wrong with the model. Usually the answer is: nothing. The model is doing exactly what it did before.
What changed is the question being asked of it. Accuracy is not a property a model carries around — it is the result of comparing a model against a specific set of labelled examples. Change the examples and you change the number, without touching a single weight. A demo dataset and a live camera feed are different sets of examples, often radically so, and the difference is almost never random.
The five places the gap comes from
In practice the divergence is boringly consistent. The same handful of variables account for most of it, on most sites, across most use cases.
Class imbalance makes the headline number meaningless
The fourth row deserves separate attention, because it is the one that quietly invalidates the figure everyone quotes.
Most detections worth paying for are rare. Someone without a helmet, a defect on a line, an item left behind, a vehicle going the wrong way — these are, thankfully, exceptions. If an event appears in a tiny fraction of frames, then a model that never reports it at all is still right about almost every frame it sees. Its overall accuracy will look excellent. Its usefulness is zero.
This is why we will not sign up to a single accuracy percentage as an acceptance criterion, and why you should be wary of anyone who will. The pair that actually describes behaviour is recall — what share of real events did the system catch — and precision — what share of its alerts were real. They trade against each other, and the balance is a business decision, not a technical one. A safety alert that misses nothing but cries wolf twice an hour will be muted within a week; a defect check that never raises a false alarm but misses a third of defects is quietly useless. Which error you prefer depends entirely on what happens after the alert, which is a conversation about people and process rather than about models.
What we look at before quoting anything
Because four of the five variables are physical, the first useful engagement on a vision project is not a model selection exercise. It is a look at the cameras.
We want to know where each camera is mounted and at what angle, what the light does across a full day including the switch to artificial or infrared, what sits between the camera and the subject at busy times, how the scene changes between shifts, and what the frame rate and resolution actually are at the point the stream arrives rather than what the camera is capable of. Then we want labelled footage from those cameras, covering the awkward conditions on purpose rather than the convenient ones.
Only after that is it sensible to talk about numbers, and even then they should be stated per class and per condition. An aggregate figure averaged across a well-lit morning and a backlit evening describes neither. If you are working with cameras you already own, the constraints are tighter still — we have written separately about adding analytics to CCTV already installed, where the camera positions are a given rather than a decision.
The accuracy you can buy with a ladder
The encouraging part is how much is recoverable without retraining anything. Moving a camera, changing its angle, fitting a different lens, adding a light, or raising the frame rate will each often do more than a model change would, and can be tried in an afternoon rather than a sprint.
The same is true of the rule layer that sits between detection and alert. Narrowing the zone the system is asked to watch, requiring a condition to persist for a minimum duration before it counts, and deduplicating repeated alerts about the same subject all change the numbers an operations team experiences, without touching the model at all. We went into that layer in detail in PPE detection on a factory floor, because it is usually where these systems are won or lost.
The ordering we work to is: fix the physical, then fix the rules, then consider the model. It is the cheapest sequence and, more often than teams expect, it stops before the third step.
Accepting a system without a number you both trust
The way to avoid the awkward meeting entirely is to never quote a figure that came from someone else's data. Agree the acceptance test up front, on footage from the cameras that will run the system, labelled by someone who knows the site, with precision and recall reported separately per class and the preferred failure mode written down.
And then plan for it to move. Sites are not static: seasonal light shifts, a camera gets nudged during maintenance, a new product or a changed uniform appears, scaffolding goes up for a month. These arrive gradually and announce nothing, so performance degrades quietly rather than breaking visibly. An accepted system needs periodic sampling against current footage and an agreed trigger for retraining — otherwise the number you accepted slowly stops being true, and nobody finds out until someone stops trusting the alerts.
Questions we get asked
Why does a model perform worse after deployment?
Usually it does not — the model is unchanged, the data is not. An accuracy figure measures a model against one dataset, and demo datasets are curated: even light, clear sightlines, sensible angles, balanced examples. The original number never described your site. Measure on your own footage before anyone commits to a figure.
Why is a single accuracy percentage misleading?
Because useful detections are rare. If an event appears in a small fraction of frames, a model that never reports it still scores highly — it is right about every ordinary frame. Precision and recall are the pair that describe behaviour, and one number cannot carry both.
What has the biggest effect in a real deployment?
The physical variables outrank the algorithmic ones, and lighting usually tops the list — backlighting at certain hours, the switch to infrared, mixed daylight and artificial light. Camera angle is second: a camera at 4m steeply oblique sees a different shape than near-eye-level training footage. Both are fixable with a site survey.
How should accuracy be measured before acceptance?
On footage from the cameras that will run it, labelled by someone who knows the site, across the conditions that matter — multiple times of day including the light transition, multiple shifts, and the awkward cases deliberately included. Report precision and recall per class, and agree in advance which matters more.
Can accuracy improve without changing the model?
Very often, and more cheaply. Moving a camera, changing the angle, adding light, a different lens, a higher frame rate. So can changing what is being decided: a narrower zone, a minimum dwell before alerting, deduplication of repeat alerts. Physical and rule-layer fixes come before algorithmic ones.
Does accuracy stay stable after acceptance?
No. Seasonal light, a camera knocked during maintenance, a new product line or uniform, scaffolding — these shift things gradually and trigger no alarm, so performance degrades quietly. Plan periodic labelled sampling against current footage and a defined retraining trigger.
We build computer vision systems that run on-premise, on the cameras a site already has. If you are being quoted an accuracy figure, the question worth asking is which dataset it was measured on — and whether anyone has looked at your cameras yet.