Most on-premise LLM plans size the hardware from the model name, which is the wrong variable. Weights are a fixed cost you can calculate in one line; the cost that moves is the KV cache, which grows with context length times concurrency. Quantisation does not mainly buy you a bigger model — it buys you more simultaneous users on the same card. And the work that actually consumes the project is everything around the model: access control, audit logging, versioning, evaluation and somebody to own it.
The decision to self-host usually gets made for good reasons — data that cannot leave the building, a regulator with opinions, or a cloud bill that grows with every success. We have written about whether that decision makes sense on cost. This is about what happens after you have made it.
The short answer is that running the model is easy and running it as a service is not. Almost every surprise in these projects comes from one of two places: GPU memory behaving differently than expected, and the realisation that the model was the smallest part of the build.
Where the GPU memory actually goes
Memory sizing is the first thing teams get wrong, usually by reasoning from the model's parameter count and stopping there.
The weights are straightforward. A model's weight footprint is approximately parameters × bytes per parameter. At 16-bit precision that is two bytes each, so a 7-billion-parameter model needs roughly 14 GB just to load. Useful, and entirely predictable.
The part that moves is the KV cache — the attention keys and values held for every token currently in play. It grows with the length of the context and with the number of requests being served at once. One user with a short prompt barely registers. Forty users each holding a long document in context is a different machine entirely.
This is why the same model on the same card can serve a handful of concurrent users or a great many, depending entirely on decisions nobody wrote in the spec.
Two practical consequences follow.
First, decide your context budget early, because it is a hardware decision wearing a product costume. "Let users paste in whole contracts" and "answer short questions over retrieved snippets" are different purchases.
Second, retrieval is a memory optimisation as well as an accuracy one. Pulling the three relevant paragraphs instead of the whole document improves answers and costs a fraction of the KV cache. Teams reach for longer context when better retrieval would have been cheaper in both senses.
What quantisation is really for
Quantisation stores weights at lower precision — 8-bit or 4-bit instead of 16. Dropping to 4-bit cuts the weight footprint to roughly a quarter.
The instinct is to spend that headroom on a larger model. Usually the better trade is to spend it on concurrency: the freed memory becomes KV cache, and the same card serves considerably more simultaneous users.
There is a quality cost, and its size depends on the model and the task. Extraction and classification tend to tolerate quantisation well; tasks involving long chains of reasoning are more sensitive. This is measurable and it is worth measuring, because the answer differs enough between workloads that borrowing someone else's conclusion is unwise.
Choosing a serving stack
The decision here is simpler than the number of options suggests, and it turns on one question: how many requests at once?
- Concurrent production traffic — use a server built around continuous batching, such as vLLM or Text Generation Inference. Batching is what converts one GPU into a service for many users instead of a queue of one. Without it, you buy a GPU and leave most of it idle while requests wait.
- Single user, local development, or CPU and mixed hardware — the llama.cpp family, including Ollama, is far simpler to stand up and perfectly adequate.
Note what this does to latency. Batching improves throughput, not the latency of an individual request — and under load, a batching server holds latency steady while an unbatched one degrades badly. If you have a strict per-request latency target, state it as a number before choosing anything, because it constrains batch sizes and therefore the economics.
Setting expectations on capability
An open-weight model you host is generally not at the frontier, and projects that quietly assume otherwise end in disappointment.
But most enterprise work is not frontier work. Classification, extraction, summarisation, routing, and question answering over your own documents are well within range, and for those the gap narrows a great deal. The failure mode is scoping a flagship "ask it anything" assistant when the actual need was six well-defined tasks that a smaller model handles reliably.
Decide this with your own evaluation set — a few hundred real examples with known-good answers, drawn from your actual workload. A public leaderboard tells you about the leaderboard.
Everything that is not the model
This is where on-premise projects actually spend their time, and it is routinely left out of the estimate:
- Access control. The model must respect the same boundaries as the data it reads. A retrieval system that ignores per-user permissions is a data leak with a chat interface — and it is the single most common serious mistake in these builds.
- Audit logging. Prompts and responses, with a retention policy somebody has agreed. Note that prompts routinely contain personal and commercial data, so this is itself a governed store, not a debug log.
- Rate limiting. Finite GPU capacity means one enthusiastic team can starve everyone else.
- Versioning. Of the model and of the prompts. Behaviour will change; you need to know what changed and when.
- An evaluation gate. Run the eval set before any model or prompt swap. Without it, upgrades are a coin toss you perform in production.
- A rollback path. Keep the previous model loadable.
The question that actually decides it
Who runs this in eighteen months?
A self-hosted model is infrastructure. It needs patching, driver and library upgrades, capacity monitoring, incident response, and a plan for the day a materially better open-weight model appears. Without a named owner, it drifts — and a drifting GPU service with stale dependencies and no eval coverage becomes a liability rather than a capability.
So the honest comparison is not GPU cost against API cost. It is GPU cost plus an owner against API cost. For organisations with a real data-residency constraint that maths still comes out clearly in favour. For organisations self-hosting because it seemed cheaper, it often does not.
Questions we get asked
How much GPU memory does a self-hosted LLM need?
Weights are roughly parameters × bytes per parameter — a 7B model at 16-bit is about 14 GB. But that is the fixed cost. The variable one is the KV cache, which grows with context length × concurrency, which is why the same model on the same card can serve very different numbers of users.
What does quantisation buy you?
Lower-precision weights — 4-bit cuts the weight footprint to roughly a quarter. The headroom is usually best spent on concurrency rather than a bigger model, since it becomes KV cache. There is a quality cost that varies by model and task, so measure it on your workload.
Which serving stack should we use?
For concurrent production traffic, something with continuous batching — vLLM or Text Generation Inference. For single-user, local or CPU workloads, the llama.cpp family including Ollama is simpler. The deciding question is how many simultaneous requests you need, not which project is most popular.
Is a self-hosted model as good as a frontier API model?
Usually not at the top of the capability range. But most enterprise workloads — classification, extraction, summarisation, routing, QA over your own documents — sit well within reach of current open-weight models. Test on your own evaluation set rather than a leaderboard.
What else does an enterprise deployment need?
Access control that matches the data's own boundaries, audit logging with an agreed retention policy, rate limiting, model and prompt versioning, an evaluation gate before any swap, and a rollback path. Teams budget the GPU and forget these, which is where the effort actually goes.
Who runs it once it is live?
The question that decides whether self-hosting is wise. It needs patching, upgrades, capacity monitoring, incident response and a model-update plan. The honest comparison is GPU cost plus an owner against API cost.
We build AI systems that run on the client's own infrastructure, including the access control, logging and evaluation scaffolding around them. If you are sizing a deployment, the two numbers worth settling first are your context budget and your expected concurrency — almost everything else follows from them.