Choosing a local inference runtime: llama.cpp, Ollama, Krasis and the alternatives

Choosing a local model is only part of a private AI deployment. The software running it determines which hardware you can use, how requests share that hardware, and how much work it takes to keep the service available.

The choice depends on what you need to run. A desktop assistant, a shared document service and a large coding model on a single GPU present different problems. The most useful comparison starts with those requirements.

About this comparison: This guide draws on the projects’ published documentation, reviewed on 19 October 2026. The suggested starting points are our engineering judgement. This is a selection guide; we have not run a controlled benchmark across all the products below.

Northloom develops Krasis and offers private AI architecture reviews. Building a runtime informs the questions we ask about model support, memory and execution paths; the right choice still depends on your workload and may be another project. See our inference engineering work.

The options at a glance

Some of these products include a user interface and model management; others primarily provide an execution engine or serving framework. Their capabilities overlap, and a desktop application may use one of the listed engines underneath.

llama.cpp: a useful baseline across different hardware

llama.cpp provides C/C++ inference, quantised models and CPU/GPU execution across a broad range of hardware. Its backends include CUDA, Metal, HIP and Vulkan. It also provides an API server, so it can support an application as well as interactive local use.

It is a sensible baseline when you want GGUF model files, explicit settings and flexibility about hardware. CPU/GPU hybrid execution can help run a model that exceeds VRAM, although fitting it does not establish that it will be responsive enough.

The trade-off is configuration work. Model support, offload choices and performance depend on the backend and build. Keep those details with your test results. Two machines running something called “llama.cpp” may be using materially different execution paths.

Ollama: convenient model management and integration

Ollama makes it straightforward to download models, run them and connect applications to a local service. It is a useful starting point for developers and small internal deployments where simple setup has real value.

Its convenience does not remove memory planning. The Ollama FAQ describes model loading, CPU/GPU placement, queueing and parallel requests. Context length and parallelism affect memory requirements; loading different models can also change the delay users experience.

Test the application through Ollama itself. Do not assume that an underlying engine’s benchmark predicts the behaviour of the complete service. For a private deployment, also check the selected model’s location and disable cloud features where required; the FAQ documents a local-only setting.

LM Studio: explore models through a desktop interface

LM Studio combines model discovery, downloads, chat and local API serving. It uses llama.cpp for GGUF models and also supports MLX on Apple silicon. Its tooling includes headless operation through llmster.

This makes it useful for exploring candidate models with staff who prefer a graphical interface, then testing an application against a local endpoint. A desktop interface and a server deployment can therefore form parts of the same evaluation.

Record the selected engine version, model and settings before treating an experiment as a repeatable service. Compare the operational route you intend to use, including startup, updates and monitoring. A successful conversation in the desktop app is the beginning of that evaluation.

MLX LM: a direct option for Apple silicon

MLX LM provides text generation, quantisation and fine-tuning tools built on MLX for Apple silicon. It is worth including when a Mac is the proposed inference machine and you want direct command-line or Python integration.

Unified memory changes the hardware calculation, but the operating system, applications, model and inference state still share a finite resource. Test representative long prompts while the rest of the intended workload is running.

The project documents prompt caching and controls for memory use during long inputs. Measure their effects with your workload. A result from a short prompt on an otherwise idle Mac does not establish capacity for a shared service.

vLLM: evaluate it for a shared inference service

vLLM provides serving features including continuous batching, paged attention memory management, prefix caching and distributed execution. It supports a range of hardware and model architectures, subject to the relevant platform and model requirements.

It belongs on the shortlist when several users or applications will share inference capacity. In that setting, the question extends beyond one response’s generation speed to how much useful work finishes within an acceptable wait.

Measure queue time, time to first token and completion latency under load. Check the exact model, quantisation and hardware support before designing around a feature. A well-performing shared service still needs enough memory for its model, inference state and concurrent requests.

SGLang: another candidate for demanding serving workloads

SGLang is an inference framework covering language and multimodal models, with a focus that includes agent workloads and large-scale serving. Its documentation provides model and hardware deployment recipes.

Consider it alongside vLLM when operating a shared service, especially if a documented recipe closely matches your proposed deployment. Include realistic agent conversations where those are part of the requirement: repeated context, tool definitions, tool results and long outputs.

Choose between serving frameworks using those results and your ability to operate them. Installation, recovery, observability and version upgrades all contribute to the cost of a service. A favourable headline benchmark cannot answer those questions for your deployment.

Krasis: large MoE models on constrained NVIDIA hardware

Krasis is Northloom’s runtime for large mixture-of-experts models on NVIDIA GPUs. A MoE model activates selected expert networks for each token; its total weight storage can be much larger than the portion used at one step.

Krasis manages expert residency between system RAM and VRAM, with GPU prompt processing and decode. Its performance-sensitive serving path uses Rust and CUDA. Memory budgeting uses runtime measurements to inform how much can remain resident as prompt requirements change.

Consider it when a supported large MoE model is useful for the task but will not fit entirely in GPU memory. Allow for substantial system RAM, source weights, generated caches and first-run preparation. Check the published quality results and model-specific limitations as well as speed. We would not make it the default choice for every local deployment.

Strata: a focused consumer-hardware route

The Strata project compared here is Niko1221/Strata. Several unrelated projects share the name. Its current documentation centres on Qwen3.8-Flash-Next, with Windows/Linux setup, local APIs and NVIDIA/AMD hardware paths. Other platforms are documented as experimental.

That focus can simplify getting one particular model running. It also makes model choice an important part of the comparison: its documentation lists different quantisations and variants, including a coding variant with experts removed.

Keep those distinctions in your results. A smaller or modified checkpoint may be a good choice if it meets the task, but comparing it with a different checkpoint measures more than the runtime. Check platform-specific feature limits and test the chosen configuration’s output quality, context behaviour and concurrent request handling.

FreeToken: CPU and GPU co-execution for MoE models

FreeToken targets large MoE models on consumer hardware. The project describes bandwidth-adaptive CPU/GPU co-execution, expert caching, streamed prompt processing and dynamic allocation between expert and context memory. It offers desktop and command-line entry points.

Include it when evaluating a supported MoE model whose weights exceed VRAM. Its design makes the CPU, system memory and interconnect part of the inference platform, so a GPU specification alone is an incomplete description of the machine.

Check the supported checkpoint and weight preparation route, then measure the whole workflow. Repeat with long inputs and subsequent agent turns if relevant. Treat the project’s performance claims as results to investigate under their stated conditions, rather than a promise for another machine.

Build a shortlist around your workload

Our suggested starting points are:

• Desktop exploration: Ollama or LM Studio, with llama.cpp when direct control is useful.
• Apple silicon: include MLX LM and a llama.cpp-based configuration in the evaluation.
• A shared application: include vLLM and SGLang, then compare capacity, latency and operating effort.
• A large MoE model beyond VRAM: include Krasis, FreeToken and, where its model focus fits, Strata. Keep a suitable llama.cpp configuration as a baseline.

These are overlapping shortlists. A small shared workload might run comfortably through Ollama; a specialist engine might be unnecessary if a smaller model already meets the requirement. Start with the least complicated configuration that passes your acceptance tests.

Compare completed work, memory and quality

A useful trial records:

1. The exact configuration. Runtime release or commit, model revision, weight precision, context settings, CPU, GPU, RAM and operating system. Record any pruning, context truncation or speculative decoding.
2. The real inputs. Short and long prompts, fresh and reused context, and the tool schemas or images your application needs. Separate cold starts from warm requests.
3. User-visible performance. Time to first token, complete task time and latency under expected concurrency. Keep prompt processing and generation rates separate.
4. Resource requirements. Peak GPU and system memory, storage, preparation time and any dependence on disk reads during inference.
5. Useful output. Correct answers, valid structured results, successful tool calls and failure rates. Re-run the same task set after changing precision or execution settings.

For a runtime comparison, hold the model and workload constant as far as possible. When each runtime requires a different format, record how those formats were produced and assess quality alongside speed. When comparing complete deployment options, different models may be appropriate, provided each must pass the same task requirements.

An API described as compatible also needs testing with your client. Streaming, tool arguments, reasoning output and context handling can expose differences that a simple chat request misses.

Finally, review the complete deployment: runtime and model licences, network access, logs, authentication, updates and recovery. Running inference locally gives you control over its location; the surrounding application determines where documents, tool results and other data travel.

Our guides to testing model suitability and diagnosing slow inference explain how to turn these questions into an evaluation. For a proposed deployment, Northloom’s architecture review examines the runtime, hardware and application assumptions together.