What is model serving?
Model serving is the software and infrastructure that keeps a trained model loaded and ready to answer requests. A serving system loads the model weights into graphics processing unit (GPU) or CPU memory, exposes an inference endpoint on the network, converts incoming requests into the form the model expects, runs inference, and returns the result. It also handles the unglamorous work of queuing requests when traffic spikes, running several requests through the model at once, and reloading when a new model version is released.
For classic machine learning models, serving is relatively simple because each prediction is one fast computation. For large language models it is a specialized discipline. An inference server for a language model manages the KV cache that stores each conversation’s attention state, groups requests through continuous batching so the GPU stays busy, streams tokens back as they are produced, and may use quantization or speculative decoding to raise throughput. These choices determine tokens per second, time to first token, and how many users one GPU can support.
Why model serving matters
Serving is where the cost of AI is incurred and where users form their opinion of it. A model that scores well on a benchmark but is served badly feels slow and unreliable. The same model on a well-tuned inference server can handle several times the concurrent users on the same hardware, which directly changes how much compute a company must buy or rent. Serving is also the enforcement point for access control, rate limiting, and logging, since every request passes through it.
Model serving in practice
A law firm runs an open-weight language model on two GPU servers to summarize discovery documents and compare contract clauses. Attorneys submit long documents in bursts, so the serving layer batches requests, caches the shared system prompt, and streams output so a reviewer can start reading within a second. During a large matter the firm adds a third server, and the AI gateway in front spreads requests across all three. Usage logs from the serving layer show which practice groups use the system and feed the internal chargeback.
What to watch for
- Measure throughput under realistic concurrency. A vendor’s number for a single request says little about a busy afternoon.
- Match GPU memory to the model, its precision, and the context lengths users actually send, with headroom for the KV cache.
- Confirm the serving layer offers an OpenAI-compatible API if existing applications expect one, because rewriting clients is slow.
- Decide how model updates reach the server without dropping requests that are in flight.