What is compute?
Compute is the processing capacity an AI workload consumes: the processors, the hours they run, and the arithmetic they perform. Engineers measure it in floating-point operations per second (FLOPS) for raw speed, in GPU-hours for a job, and in tokens per second for a language model serving requests. A model that took a great deal of compute to train ran on thousands of GPUs for weeks.
Compute reaches hardware through a runtime, the software layer that turns a model’s math into instructions a processor executes. Deep learning frameworks such as PyTorch call a vendor runtime, most often CUDA on Nvidia GPUs, to perform the matrix operations. Inference servers such as vLLM add scheduling, batching, and memory management so one GPU can serve many users at once. The runtime determines which hardware a model can run on and how much of its speed it actually uses.
Training compute compared with inference compute
Training and inference place very different demands on hardware. Training a model from scratch processes an entire dataset many times, computing gradients and updating billions of parameters at every step. It needs many GPUs with fast interconnects, large memory per GPU, and days or weeks of uninterrupted time. Fine-tuning an existing model is far cheaper, and parameter-efficient methods such as LoRA bring it within reach of one GPU for a few hours.
Inference, producing an answer from a trained model, runs one forward pass per token generated. Each request is small, but requests arrive all day, so the requirement is sustained capacity rather than a burst. Inference is limited by GPU memory and memory bandwidth more than by raw arithmetic. For most mid-sized companies, nearly all compute spending goes to inference, because they adopt existing models rather than training their own.
Compute in practice
A third-party logistics company builds a model that predicts late shipments. Training on three years of shipment history takes a cloud GPU instance for two days each quarter. Scoring every active shipment every fifteen minutes is the inference side, and it runs on a modest CPU server because the model is a gradient-boosted tree. When the company later adds a language model that drafts customer delay notices, inference moves to a single owned GPU, which handles the day’s volume with capacity to spare.
What to watch for
- Separate training and inference in every estimate, because the hardware, duration, and cost pattern differ completely.
- Check that the runtime supports the hardware being bought, since a model tuned for one vendor’s GPUs may run slowly or not at all on another’s.
- Measure utilization after deployment, because GPUs sitting at low load are the most common form of wasted compute.