What is a graphics processing unit (GPU)?
A graphics processing unit (GPU) is a processor designed to perform thousands of simple arithmetic operations in parallel. It was built to draw pixels for video games, where every pixel needs the same calculation at the same time. Neural networks have the same shape of work: multiplying large matrices of numbers, where each cell can be computed independently. A GPU does this many times faster than a central processing unit (CPU), which is built to run a few complex tasks very quickly in sequence.
A modern data center GPU combines thousands of small cores, dedicated units for matrix math, and a block of very fast memory on one card. Memory size and memory bandwidth decide how large a model the card can run and how fast it answers. Several GPUs in one server are linked by high-speed interconnects so they can share a model too large for any single card.
Why GPUs matter
Almost every AI capability a business buys or builds depends on GPUs somewhere. Training a large language model takes thousands of them for weeks. Serving one to a company’s staff takes between one and eight, depending on model size and concurrent users. The market is dominated by Nvidia, whose CUDA software most AI frameworks target first, with AMD as the main alternative, and the choice shapes which models and inference servers run without extra engineering.
GPUs are also the scarce and expensive part of an AI budget. They draw several hundred watts each, need serious cooling, and are hard to obtain when demand spikes. Whether to rent cloud GPUs or buy servers, and how many, turns on this one component.
GPU in practice
A property insurer wants a language model to summarize claim files and draft adjuster correspondence for two hundred staff. The team selects an open-weight model of about thirty billion parameters, quantizes it to eight bits, and finds it fits in one data center GPU with room for the staff’s concurrent conversations. One server with one GPU handles the entire department. The fraud scoring model that flags suspicious claims runs on the insurer’s existing CPU servers, because it is a gradient-boosted tree.
What to watch for
- Memory size decides which models fit, and it matters more than raw speed for most business inference workloads.
- Consumer gaming cards lack the memory, reliability features, and licensing terms for data center use, even though they run the same software.
- Check that the chosen inference server and model format support the GPU vendor before buying.
- Plan power, cooling, and rack space early, because a GPU server demands far more than the application servers already in the room.