How much GPU memory does a local LLM actually need?

A local LLM needs memory for its weights, the state of active requests and the runtime’s working allocations. The model’s download size answers only part of the question. To choose a GPU, you also need the intended context length, number of simultaneous requests and execution settings.

An 8-billion-parameter model at four bits per weight has about 4 GB of raw weights. A 70-billion-parameter model at the same precision has about 35 GB. Neither number is a complete VRAM requirement, and neither tells you how quickly the model will complete your work.

About the examples: these are calculations for planning, not measurements of a particular model or a guarantee that a GPU will fit it. Documentation was reviewed on 19 October 2026.

Northloom develops Krasis, which addresses large-model inference with limited GPU memory, and offers private AI hardware planning. Our approach distinguishes resident weights, request state and peak working allocations, then measures the intended configuration before making a sizing decision. See our inference engineering work.

Start with the weights

For a model stored at one precision, the raw weight calculation is:

Weight bytes = parameter count × bits per weight ÷ 8.

The table uses decimal billions of parameters and decimal GB. Hardware tools may report GiB instead: 1 GiB is 1,073,741,824 bytes, while 1 GB is 1,000,000,000 bytes.

Quantised formats also store scales and other metadata. Some tensors may retain a higher precision, and a runtime may create a different representation when loading. Use the actual checkpoint and runtime allocations to refine this estimate. The Transformers quantisation documentation illustrates how precision and offloading choices affect memory use.

Reducing precision also changes the configuration you need to evaluate. Confirm that the quantised model still passes your task tests before using its smaller size to justify a hardware purchase.

Add the context memory

Many autoregressive transformer models store keys and values from earlier tokens in a KV cache. This lets the model reuse previous work while generating its next token. The cache can become a substantial allocation for long documents and conversations.

For a conventional transformer with full attention in every layer, a useful estimate is:

KV cache bytes = 2 × layers × KV heads × head dimension × cached tokens × bytes per cache value.

The factor of two covers keys and values. Use the number of KV heads, which may differ from the number of query heads. Grouped-query attention reduces that number; NVIDIA’s attention documentation explains the distinction.

Consider an illustrative architecture with 32 layers, 8 KV heads, a head dimension of 128 and a 16-bit cache. Its cache uses:

2 × 32 × 8 × 128 × 2 = 131,072 bytes per token.

At 8,192 cached tokens, that is 1 GiB per sequence. At 32,768 tokens, it is 4 GiB. Four independent sequences of that longer length would require about 16 GiB for their caches alone, before allocator overhead or possible sharing.

Count the complete sequence: instructions, conversation history, retrieved documents, tool definitions and results, plus space for the generated answer. The advertised maximum context is a model capability; your deployment still needs the memory and processing capacity to use it.

This formula does not describe every architecture. Sliding-window attention, compressed attention, recurrent state and hybrid models need different calculations. Static allocation, paging and prefix sharing can also change actual usage. The Transformers cache guide describes cache strategies and their memory trade-offs.

Allow for working memory and the busiest phase

A runtime needs more than weights and cache. Activations, temporary buffers, kernels, device contexts and execution graphs can all consume memory. Other applications may already occupy part of the GPU.

The peak may occur while processing a long input rather than while generating the answer. Measuring an idle model immediately after loading can therefore give a misleadingly comfortable result.

For an illustrative budget, suppose your chosen configuration has 5 GiB of resident weights, a 4 GiB cache and 3 GiB of other peak allocations. It already needs 12 GiB before additional operating headroom. A nominal 12 GB card is not a sound purchase on that arithmetic. Those allocations must be measured on the intended configuration; a fixed percentage added to download size cannot establish them.

Concurrent users change the calculation

One hundred staff do not necessarily mean one hundred simultaneous requests. Measure the demand pattern: a background extraction service, an interactive assistant and an agent running several tasks can place very different loads on the same model.

Weights can normally be shared by requests using one loaded model. Independent request state still needs space, while batching changes working allocations. Some runtimes reserve capacity in advance and others allocate it as requests grow.

The Ollama FAQ, for example, documents how parallel requests and context settings affect memory. Test the runtime settings your application will actually use. Queueing requests can reduce simultaneous memory demand, but users then wait longer.

What if the model is larger than VRAM?

CPU offloading and specialist runtimes can keep some weights in system RAM while using the GPU for selected work. This can make a larger model usable, but speed then depends on memory bandwidth, CPU execution, transfers and the runtime’s strategy.

For mixture-of-experts models, distinguish total parameters from active parameters. The active count describes the selected computation at a step; the full checkpoint still needs storage somewhere. Expert caching can change which weights reside in VRAM without making the remaining weights disappear.

Northloom’s Krasis addresses this large-MoE use case. Our runtime comparison covers it alongside alternatives. Choose using supported models, output quality and measured task completion time.

Multiple GPUs also need a supported distribution strategy. Two cards do not automatically behave as one larger card: each must fit its assigned weights, state and buffers, and communication affects performance. On Apple silicon, unified memory is shared with the operating system and other applications, so installed RAM is not all available for inference.

Verify the requirement before buying

Ask for a test using the exact checkpoint, quantisation, runtime version and settings you intend to deploy. Include:

1. A typical request and your longest supported input, each with a realistic output allowance.
2. Independent requests at expected peak concurrency, including fresh context.
3. Startup, prompt processing, generation and repeated turns.
4. Peak GPU and system memory, together with latency and output quality.
5. Behaviour when demand exceeds capacity: queueing, rejection or failure.

Record usable free memory before the run and the lowest free memory during it. Agree explicit headroom for the deployment, then repeat after material configuration changes. Our hardware-buying guide explains how to turn those results into a supplier requirement.

The purchasing question is whether the proposed system can complete your workload at an acceptable speed with room to operate reliably. Northloom’s infrastructure planning assessment helps connect those requirements to a hardware decision before you commit.