What is GPU memory (VRAM)?
GPU memory, also called VRAM (video random-access memory), is the high-speed memory built onto a graphics card. During inference the whole model must sit in this memory, along with the working data for every conversation being served. If the model does not fit, it does not run, however fast the card’s processors are. Data center GPUs carry far more memory than consumer cards for this reason.
The arithmetic is simple. A model’s weights occupy its parameter count multiplied by the bytes stored per parameter. At 16-bit precision each parameter takes two bytes, so a 7-billion-parameter model needs about 14 GB and a 70-billion-parameter model needs about 140 GB. Quantization to 8-bit halves that, and 4-bit halves it again. A 30-billion-parameter model at 8-bit needs roughly 30 GB for weights, which leaves room on an 80 GB card. At 16-bit it needs 60 GB, and the card is nearly full before a single user connects.
How GPU memory works
Weights are only the first claim on memory. The KV cache stores the attention keys and values for every token in every active conversation, so it grows with the context window and with the number of simultaneous users. A long document pasted into a prompt can consume several gigabytes of cache on its own. Activations and the inference server’s own buffers take a further share. The practical capacity of a card is therefore the memory left after the weights are loaded, divided by the cache each concurrent conversation consumes.
Why GPU memory matters
Memory is the number that turns a model wish list into a hardware order. A legal department that wants a 70-billion-parameter model to review long contracts needs several data center cards or aggressive quantization, and must choose between those compromises. A smaller firm serving a 12-billion-parameter model to thirty staff can do it on one mid-range card. The same reasoning sets the limit on concurrent users. Once the cache is full, new requests queue and response times climb.
Kirk Tech Solutions typically sizes GPU memory from the KV cache measured at the expected concurrency and real prompt lengths, since estimates from the weight count alone run optimistic.
What to watch for
- Ask for the memory requirement at the intended precision and context length, since a model’s headline size describes weights only.
- Count peak concurrent conversations, because each one holds cache that is released only when it ends.
- Prefer a card with spare memory over a faster card that is full, because headroom is what absorbs long prompts and traffic spikes.
- Check that the inference server supports paged or shared caching, which stretches the same memory across more users.