What is model size?
Model size is a measure of how large a model is, stated as its parameter count or as the gigabytes those parameters occupy. A model with eight billion parameters stored at 16-bit precision is about sixteen gigabytes on disk and needs at least that much memory on a graphics processing unit (GPU) to run, plus working space for the KV (key-value) cache, which grows with conversation length. Quantization to 8-bit or 4-bit storage roughly halves or quarters the footprint with a modest loss in quality.
Size sets the hardware tier. Models up to about eight billion parameters run on a single workstation GPU and, when quantized, on a laptop. Models in the thirty-billion range fit on one data center GPU. Models of seventy billion and above need multiple GPUs or aggressive quantization, and frontier models need a rack of them.
Why model size matters
Model size drives what private AI costs to run, through the GPU bill, and how fast responses come back, because every generated token requires reading all the weights once. For a company choosing a model to host, the practical question is the smallest model that passes its own evaluation. A credit union’s loan-document assistant on a thirty-billion-parameter model that fits one GPU is cheaper to own and faster than the same assistant on a model twice the size that scores only slightly better on the credit union’s test set. FlatClaw, the open-source private AI platform from Kirk Tech Solutions, uses this single-GPU class of model as its standard configuration.