A file upload is a two-line form that works first time, which is exactly why it gets no estimate. Then production arrives: four separate size ceilings held in four different configurations and all reported to the browser identically, an upstream link roughly a tenth of the one it was built on, and a single long request that restarts from zero on any hiccup. The bytes are the easy half. The harder half is the processing state nobody designed, which is also where the orphaned files come from.
Almost every business application grows a file upload. A claim needs a photograph, an application needs a passport scan, a report needs last quarter's spreadsheet. It is the feature least likely to appear in an estimate, because the first version takes an afternoon and works.
It works because it was tested with a two-megabyte file on office fibre. Neither of those is what production looks like.
Four ceilings, one error message
Between the browser and your code there are several places a request body gets measured, and in a typical deployment each lives in a different configuration file owned by a different person.
There is a limit at the CDN or reverse proxy in front. There is one at the web server. There is one in the application runtime — and in some runtimes it is two values, one for the body and one per file. And there is an idle or request timeout at the load balancer that will close the connection partway through a slow upload regardless of any size setting.
Raise one of them and the upload still fails, at the next one. Worse, the failures are hard to tell apart from the browser: the connection closes, and what the user sees is a request that did not complete. The diagnostic path runs through four sets of logs that may not all be available to the same team.
Two habits make this bearable. Decide the maximum file size as a product decision, write it in one place, and configure every hop from that number. Then state it in the interface and enforce it client-side first, so the common case is a clear message before anything is sent rather than a dead request four minutes in.
The connection nobody measured
Office and data-centre connections are fast in both directions. Consumer and mobile connections are not: the upload direction is routinely a small fraction of the download direction, which is why a video call degrades when someone else in the house starts a backup.
The practical consequence is that a file which uploaded in eight seconds during testing can take several minutes for a customer at home, and longer on a phone with two bars. Nothing is broken. The estimate for “upload a file” was simply made against the wrong link, in the same way that performance assumptions get made against the laptop the code was written on.
Which leads to the part that actually decides whether uploads succeed.
One request is the wrong shape
A plain form post sends the whole file in a single request, which means one connection held open for the entire transfer. On a slow upstream link that is minutes. Any interruption — a lift, a handover between cells, a laptop lid — and it starts again from nothing.
Users do not read that as a network problem. They read it as an application that cannot accept a file, and after the second attempt they email it to somebody instead.
Splitting the file into parts fixes the shape of the problem rather than its symptoms. A failure now costs one part rather than the whole transfer, progress reporting becomes honest because parts are genuinely confirmed, and a resumed upload is possible. Mature client libraries exist for this in every ecosystem; writing it from scratch is rarely the right call.
The related change is where the bytes go. Rather than streaming through your application, the application issues a short-lived signed URL and the browser sends parts straight to object storage. Your servers stop needing memory, disk and a held-open request for every concurrent upload, and they keep the parts they should own: who is allowed to upload, what the cap is, and recording that the object arrived.
What is in the file
Once the bytes land, the actual work starts, and this is the stage that is usually missing from the design.
The type is not what the name says. A file extension is a user-supplied string and the declared content type is a browser-supplied one. Determine the real type from the file's own content and check it against what you accept, because the alternative is storing and serving something you did not intend to.
Size is not the same as cost. A four-megabyte PDF can be three hundred scanned pages. If anything downstream renders, parses or extracts from it, the limit that matters is not only bytes — it is pages, duration, or pixel dimensions, which is a familiar constraint in any document extraction pipeline.
Photographs carry more than the photograph. Phone images routinely embed location and device details. If a customer uploads a picture to a claim and you later serve that file back to a third party, you have shipped their home coordinates. Strip metadata on ingest unless you have a stated reason to keep it.
Scanning is a queue, not a function call. Malware scanning takes time, which means there is a period where a file exists and is not yet cleared. That has to be a real state in your data and a real state in the interface. Showing a tick the moment the bytes arrive is a promise you have not kept.
The orphans
The quiet data problem is that an upload writes two things — an object in storage and a row in a database — through two operations that can fail independently.
Fail one way and you have objects nothing points to, paying for storage indefinitely and invisible in every report. Fail the other and you have records pointing at files that never arrived, which surface as a broken link the first time somebody clicks.
The ordinary fix is an explicit status on every file record — pending, stored, scanned, ready, failed — and a scheduled job that compares storage against records in both directions. It is a modest amount of work at the start and an archaeology project two years later.
A sensible default
- Pick a maximum size as a product decision, state it in the interface, and configure every hop from that one number.
- Upload direct to object storage with short-lived signed URLs; keep authorisation and record-keeping in the application.
- Use chunked, resumable upload for anything beyond a few tens of megabytes, with a client library rather than your own.
- Detect type from content, strip image metadata, and cap pages or duration as well as bytes.
- Give every file an explicit status and reconcile storage against records on a schedule.
- Test on a throttled asymmetric connection with a file at your stated limit. This is the test that finds all four ceilings in one go.
None of this is difficult. It is simply more than an afternoon, and the gap between those two numbers is why uploads have a reputation for being the feature that goes wrong late.
Questions we get asked
Why does an upload that works in testing fail in production?
Usually because testing used a small file on a fast symmetric connection and production uses a large one on a consumer link. Two things change at once. The request now runs long enough to meet a timeout or a body-size ceiling somewhere in the path, and the upload direction of a typical home or mobile connection is a fraction of the download direction, so the transfer takes many times longer than the test suggested.
How many size limits are there really?
In a normal deployment, at least four, each in a different configuration file and often a different team's responsibility: the CDN or proxy in front, the web server, the application runtime, and any load balancer idle timeout that ends the connection before the body finishes. Raising one and not the others produces a failure that looks fixed in staging and is not fixed in production. Find all four and write the number down in one place.
Is chunked or resumable upload worth the complexity?
Past roughly a few tens of megabytes on consumer connections, it is what makes the upload finish at all. A single request holds one connection open for the whole transfer, so a brief network interruption restarts from zero and a user on a train may never succeed. Splitting the file into parts means a failure costs one part, and it gives you honest progress and the ability to resume after a closed laptop.
Should files go through the application server or straight to storage?
Straight to storage, in almost every case. The application issues a short-lived signed URL and the browser sends the bytes directly to object storage, so your servers never hold the file, never need the memory or disk for it, and never have a request occupied for minutes. The application keeps the parts it should own: who may upload, what the limits are, and recording that the object arrived.
What has to happen after the bytes arrive?
Determine the real type from the file's content rather than its name or its declared type, check it against what you accept, scan it, and then do whatever conversion the product needs. All of that is queued work, which means an uploaded file has a status for a while and the interface has to show that honestly rather than implying the file is ready. The upload is the first half of the job and usually the easier half.
What is the most common data problem with uploads?
Orphans in both directions, because the object and its database row are written by two separate operations that can fail independently. You end up with stored files no record points to, which quietly cost money forever, and records pointing at objects that never arrived, which break on the first click. An explicit status field plus a scheduled reconciliation job that compares storage against records is the ordinary fix, and it is far easier to add at the start.
If your application accepts files and nobody has tried a large one on a slow connection, that is the afternoon worth spending. It is a standard part of how we build web applications, because it is cheaper to design than to retrofit.