What is a central processing unit (CPU)?
A central processing unit (CPU) is the general-purpose processor that runs the operating system and ordinary software on every server, laptop, and phone. It has a few fast cores, each designed to execute complex instructions one after another with very little delay. A GPU, by contrast, has thousands of slower cores that perform the same simple operation on many values at once.
In an AI system the CPU does everything except the heavy matrix math: it runs the application, authenticates users, tokenizes text, queries the database and vector store, and moves data to and from the GPU. It also runs classic machine learning models, such as regression and tree-based models, which need no GPU.
CPU in practice
A credit union scores every loan account nightly for delinquency risk with a gradient-boosted tree, and an ordinary CPU server scores half a million accounts in minutes. The same server runs the retrieval and application layers for the credit union’s language model assistant, while a GPU in another server handles only the model. Small quantized language models can also run on CPUs for low-volume tasks, with responses taking seconds.
What to watch for
- A GPU server still needs enough CPU cores and system memory to feed it, or the GPU waits idle between requests.
- Run classic machine learning on CPUs before buying GPUs, since most tabular business problems never need one.
- CPU inference of language models works for a handful of users and fails at scale, so test at the expected concurrency.