A large Mixture-of-Experts model can need hundreds of gigabytes of weight storage while using only a small fraction of its parameters for each token. That creates an opportunity: keep the model in system RAM, use limited GPU memory strategically, and move or execute the required experts as the request progresses.
It also creates a trap. A model can load, generate fluent text and still be too slow for an IDE, exhaust memory on a long prompt, or fail the work it appeared to understand.
Northloom develops Krasis, a Rust/CUDA inference engine for large MoE models on memory-constrained NVIDIA hardware. This guide draws on that engineering work and our living research white paper. It explains the decisions to measure, rather than prescribing one runtime or promising that every large model will work on a consumer GPU.
Why MoE changes the local inference equation
A dense model uses its main feed-forward weights for every token. A MoE model instead routes each token to a selected subset of expert networks, alongside attention and other shared operations. Different tokens can select different experts.
The difference between total and active parameters can be substantial. The official Qwen3-Coder-Next model card lists 80 billion total parameters and 3 billion active parameters. DeepSeek-V3 lists 671 billion total and 37 billion active. These describe particular releases; DeepSeek, Qwen and GLM families contain different architectures and configurations.
The smaller active count helps explain why inference is possible without reading every expert for every generated token. It does not mean that an 80B model has the storage requirements of a 3B model. All experts must remain available, and attention, routing and data movement still cost time.
As an illustrative calculation, 400 billion parameters stored at exactly four bits each occupy 200 GB, approximately 186 GiB. That is raw weight payload only. Quantisation metadata, tensors stored at higher precision, runtime buffers, conversation state and loading allocations add to it. A machine with 200 GB of RAM would not therefore be a defensible capacity recommendation.
Use the prepared model’s actual sizes and measured peaks when planning hardware. Nominal bit width is a starting estimate.
Choose what happens when an expert is outside VRAM
“CPU offloading” can describe materially different execution strategies:
| Strategy | Where the expert executes | What to measure |
|---|---|---|
| Keep an expert resident | GPU | VRAM use, kernels and the fraction of requests served by resident experts |
| Fetch an expert from RAM | GPU, after transfer | Actual bytes transferred, effective PCIe bandwidth and exposed waiting time |
| Execute an expert on the CPU | CPU, exchanging inputs and outputs with the GPU | CPU kernels, RAM bandwidth, scheduling and synchronisation |
Runtimes may combine these strategies. CPU execution can avoid transferring a large weight tensor but introduces CPU work. Fetching weights retains GPU execution but makes the transfer path critical. Neither wins for every model, machine or prompt.
Caching helps when future routing reuses resident experts. Measure that reuse on representative workloads: a cache warmed by one document or coding language may behave differently on another. Record bytes as well as hit rates, especially when experts have different sizes or precision.
An expert cache should change placement and scheduling while preserving the requested computation. Dropping experts or substituting an approximate routing policy changes the model and requires its own quality evaluation.
Treat prompt processing and generation separately
Inference has two distinct phases. Prefill processes the input prompt; decode generates subsequent tokens.
During prefill, many tokens can be processed together. Across a batch, those tokens may select many different experts. GPU execution can amortise the cost of bringing weights into memory over substantial useful work. Full GPU execution does not require the entire model to remain resident simultaneously: an engine can stream bounded groups of layers through VRAM.
During decode for a single conversation, each step has much less work to share the transfer cost. Keeping useful experts resident, or efficiently executing nonresident experts elsewhere, matters much more. Multiple simultaneous requests change that balance again.
This is why a short chat benchmark can give the wrong answer for a coding agent. As an illustrative example, processing 10,000 fresh input tokens at 200 tokens per second takes 50 seconds; at 2,000 tokens per second it takes five seconds. Generation has not begun in either case. These are arithmetic examples, not Krasis benchmark claims.
Test time to first token with realistic tool definitions, repository context and conversation history. Then measure generation separately. A cache hit on a repeated prompt should not be reported as fresh-prefill performance.
PCIe bandwidth can matter more than GPU compute
When nonresident experts execute on the GPU, the host-to-device link becomes part of every affected step. A useful first calculation is:
Transfer time = bytes fetched ÷ effective transfer bandwidth.
Suppose an illustrative workload fetches 1 GB of expert weights for each generated token and achieves 25 GB/s of effective transfer bandwidth. That represents 40 milliseconds of transfer work per token, before considering other work. The numbers are hypothetical. Some transfer work may overlap computation, but only measurement shows how much remains on the critical path.
The GPU’s advertised memory bandwidth does not describe this link. Check the negotiated PCIe generation and lane width, motherboard slot wiring and contention with other devices. A second physical x16 slot may have fewer electrical lanes, and several devices may share an upstream connection.
NVIDIA’s CUDA best-practices guide explains why pinned host memory, transfer batching and overlap can matter. Asynchronous calls alone do not establish overlap: scheduling and dependencies still need verification.
Before buying a faster GPU, establish whether the current one is waiting for weights. More compute cannot remove a saturated transfer link.
System RAM needs capacity and bandwidth
RAM capacity determines whether the prepared weights and operating state fit. RAM bandwidth affects CPU expert execution and the supply of data for transfers. A CPU’s core count does not establish either.
Check the populated memory channels, sustained bandwidth under the intended workload, and memory locality on multi-socket or NUMA systems. CPU workers, copies and other services can compete for the same memory resources. Adding threads can stop helping once bandwidth is saturated.
Leave room for the operating system, conversion or loading peaks, pinned buffers and any conversations retained in RAM. Record cold preparation, normal startup and steady operation separately. A comfortable memory reading after startup does not prove the startup peak is safe.
Fast storage is valuable for model files, prepared caches and loading. It does not make disk-backed paging equivalent to resident system RAM. If paging enters the request path, measure its latency explicitly before accepting the deployment.
Context memory and expert residency compete
The GPU must accommodate more than expert weights. Attention weights, the KV cache or other architecture-specific state, activations, workspaces and transient allocations all need space.
For conventional attention, KV storage grows with retained tokens and simultaneous sequences. Its size depends on the number of relevant layers, KV heads, head dimensions and storage format. Architectures using compressed attention, recurrent state or hybrid layouts need different calculations. A generic “bytes per token” value taken from another model is unreliable.
On a constrained GPU, longer context can require evicting experts to make room for attention state. The model may still fit, yet decode becomes slower because fewer experts remain resident. Context capacity and generation speed are therefore connected.
A robust budget starts from the loaded model and backend, measures prefill and decode memory peaks, and accounts for the actual request length and concurrency. It also preserves headroom for transient allocations. Krasis uses runtime calibration to inform these decisions; its white paper describes both the approach and outstanding qualification limits.
Reducing KV precision can release space, but it changes the tested configuration. So can changing attention-weight precision. Evaluate retained quality rather than assuming every memory saving is harmless.
“Supports 500K context” should answer four questions: is that the model’s stated limit, allocated capacity, a prompt actually processed, or useful work demonstrated at that length? Those are different achievements.
Correctness is part of making the model run
Our long-context research has included a particularly misleading failure: a model describes a plausible development plan, announces an action, then finishes without performing it. At around 500,000 input tokens, fluent output and successful retrieval did not establish productive capability.
The published Krasis white paper records a GLM campaign in which simple behavioural comparisons passed through 500K, but only two of four development tasks passed; hosted behavioural references completed all four. Those hosted controls are distinct from independent source-precision numerical comparisons. They do not, by themselves, establish which arithmetic or quantisation choice caused a failure.
This is why we created Agentic Development Qualification (ADQ). It tests specific model/runtime configurations on real development tasks at long contexts, with objective task checks, numerical reference comparisons and separate operational verdicts. A BF16 comparison helps assess retained numerical behaviour; completed tasks establish usefulness. Neither replaces the other.
More precision is not an automatic solution. In a later DeepSeek sweep, every precision treatment completed the four sampled tasks, including plain INT4, while every treatment separately failed the VRAM reserve requirement. The published research retains these limits rather than presenting a task pass as complete qualification.
Conversation caching needs the same scrutiny. Compare fresh full-history processing, active continuation and restoration after displacement into RAM. Verify tool state and task completion. Controlled numerical comparisons can expose differences before the visible text changes. Krasis’s October conversation-loading repair remains unqualified in the current paper; these findings are research evidence, not a claim that all modes have passed.
A practical route from candidate model to usable service
- Define the work. Collect representative prompts, tool use, context lengths, simultaneous requests and acceptable waiting times. Decide what counts as a completed task.
- Verify exact support. Check the checkpoint, architecture, quantisation, attention/cache format, hardware and runtime version. Use a supported launcher recipe; flags from another model may select a different execution path.
- Budget the whole machine. Measure prepared weights, startup peaks, system RAM, VRAM, storage and request-time headroom. Do not size the machine from active parameters.
- Establish a baseline. Record fresh-prefill rate, engine decode rate, client time to first token and complete request latency. Keep internal engine and client measurements separate.
- Diagnose before changing hardware. Instrument transfers, CPU work, GPU kernels, completion events and memory peaks. GPU utilisation shows activity; completed work establishes progress. Disable diagnostic instrumentation for repeatable speed measurements.
- Qualify quality and state reuse. Test the actual tasks at the intended context lengths, then repeated turns, conversation restoration, cancellation and recovery. Repeat relevant checks after changing precision or execution settings.
Keep the configuration, inputs and full results with each run. The guides to runtime selection, slow inference and hardware buying expand these steps.
A smaller model that reliably finishes the work may be a better deployment than a larger one whose transfer costs, latency or failure rate exceed your requirements.
Where Northloom can help
Building Krasis means working directly on Rust scheduling, CUDA execution, expert movement, memory calibration and numerical validation. That experience informs the questions we ask about an existing runtime or a proposed machine; the recommendation can still be another engine or a smaller model.
Our Inference Performance and Optimisation Assessment investigates an agreed deployment using baseline measurements and scoped experiments. Our £950 Architecture Review examines a documented design or supplier proposal before you commit. The consultancy page lists current pricing, scope and terms.
Bring the exact model, machine specification, representative inputs and the work you need completed. Those provide a sound basis for deciding whether limited VRAM is a manageable engineering constraint for your deployment.