Can a local open-weight model replace a hosted AI assistant for company work?

A local open-weight model can replace a hosted AI assistant for some internal company work. Structured extraction, classification, answers from a controlled document set and drafts based on templates are sensible candidates to evaluate. That does not establish that the same deployment can replace a general-purpose assistant across every task your staff give it.

The useful buying question is narrower: can this configuration deliver an acceptable result for this workflow, at a cost and speed that make sense?

Northloom builds AI products, including the Krasis inference runtime, and offers model suitability assessments for private deployments. Our inference engineering work informs the practical questions here: what fits, what runs well, and whether the output is useful enough to justify operating it. This guide is a decision framework, rather than a claim that a particular model has passed your workload.

Replace a workflow before replacing an assistant

A hosted assistant is a product as well as a model. It may include document parsing, retrieval, web access, code execution, connectors and a carefully developed interface. Running model weights locally does not automatically reproduce those features.

Start with one job: turning supplier documents into reviewed records, routing support requests, answering questions about approved procedures, or preparing a standard response. Define the inputs, required output and point at which a person checks it.

A local system can be valuable even if staff retain a hosted assistant for unrelated work. Passing an extraction test tells you little about whether it can investigate an unfamiliar problem independently.

Four promising tasks, and what can go wrong

Structured extraction

Invoices, forms and correspondence contain fields that can be checked against reviewed answers. Ask the model to return a fixed schema, preserve evidence and report missing values explicitly.

Test multiple totals, conflicting dates, poor scans and values present only in footnotes. Validate types and mandatory fields in code. Valid JSON can still contain an invented account number. OCR errors need their own diagnosis.

Classification and routing

A defined set of categories makes evaluation manageable. Provide category definitions, examples and an escalation route for ambiguous or unfamiliar cases.

Check results by category. If almost every message belongs in one queue, high overall accuracy can conceal failure on rare urgent cases. Measure how often staff correct the routing and what those mistakes cost.

Questions about a bounded document set

A local model can be evaluated as part of a retrieval system that supplies relevant passages from approved documents. Require references a reviewer can inspect, and test questions whose answers are absent or contradictory.

Retrieval supplies evidence; it does not guarantee that an answer follows it. Check document versions, retrieval quality and whether cited passages support each statement. Enforce access permissions outside the model, before supplying documents to it.

Drafting from templates

Standard letters, internal summaries and support-response drafts give the model a structure and limited purpose. Supply the facts it may use, specify what must remain unchanged and retain appropriate review.

Look for invented commitments, missing qualifications and softened warnings. Measure time to an approved draft, including checking and correction, rather than generation speed alone.

Where a simple local deployment may struggle

Open-ended research, conflicting evidence, specialist terminology, very long inputs and tasks requiring many dependent steps deserve scrutiny. A plausible early mistake can affect the whole answer. These are evaluation priorities, not a blanket claim that local models cannot perform them.

Workflow structure may help: retrieve a smaller evidence set, divide a task into stages, validate outputs and route uncertain cases to a person. More stages also introduce errors and operating costs. Test the resulting system end to end.

Do not grant broad tool permissions because a model summarises documents well. Untrusted text can contain instructions aimed at redirecting it. Tool access needs enforced permissions, bounded actions and separate evaluation; model instructions alone are insufficient.

Does a larger model solve the problem?

Dense models ranging from a few billion to tens of billions of parameters offer different memory and compute trade-offs. Larger mixture-of-experts models activate selected experts for each token, but their total weights still need storage and a workable memory or offloading strategy. The Hugging Face MoE explainer describes this distinction between active computation and total model size.

Parameter count alone does not settle suitability. Training, task, language, context and configuration all matter. A smaller model that passes your extraction workflow may be the better operational choice; a larger one may justify its cost when the smaller one fails important cases.

Our GPU-memory guide helps plan capacity, and the runtime comparison explains deployment options. Neither substitutes for a quality test.

What does INT4 quantisation change?

Quantisation represents weights with fewer bits. BF16 and FP16 use 16 bits per value; INT4 weight quantisation uses roughly four bits for quantised values, plus storage for scales and other data. A complete deployment also needs working memory and a context cache. Weight precision, computation precision and cache precision are separate choices; the Transformers quantisation overview explains the approaches.

Lower memory use can make a larger model practical, but there is no universal quality penalty for every task. A study of quantised instruction-tuned models found results varied with model size, method and task, including instruction following. It does not establish the outcome for a new company workload. See the instruction-tuned quantisation evaluation.

Reasoning also needs direct checks. A separate study of quantised reasoning models evaluates mathematical, scientific and programming tasks across weight, activation and cache quantisation settings. Its findings include configurations that preserved benchmark accuracy and lower-precision settings with significant risks. Neither “four-bit always ruins reasoning” nor “four-bit is always equivalent” is a sound buying rule.

Check calculations, exceptions, multi-step conclusions and compliance with output rules. Fluent prose can survive while a critical detail changes. Compare the intended quantised configuration with a higher-precision reference where feasible, keeping other settings consistent, and score both against reviewed answers. The reference can also be wrong.

Make the replacement decision on your own examples

Use the same authorised inputs and acceptance criteria for the local and hosted workflows. Reserve unseen examples for evaluation rather than prompt development. Where products use different parsing or retrieval, record the difference: you are comparing complete workflows, not isolating model quality.

Record:

• Accepted results and errors by consequence, including plausible mistakes reviewers might miss.
• Human review and correction time per completed task.
• Handling of missing evidence, unusual documents and conflicting instructions.
• Latency and throughput at expected input lengths and concurrent demand.
• Hardware, integration and operating costs, alongside data-handling requirements.

Local processing gives you control over where inference runs, but privacy depends on ingestion, logs, backups, connectors and external calls too. Open weights do not automatically mean unrestricted commercial use; check the model’s licence.

The outcome might be replacement for one task, a supervised pilot, a split between local and hosted work, or retaining the existing service. Compare economics after quality clears the agreed bar; our local-versus-cloud cost guide sets out that calculation.

Test the documents before buying the deployment

Public leaderboards help shortlist candidates. They cannot tell you how much checking staff will need on your documents. For the evaluation method, see how to assess a model against your company’s workload.

Northloom’s Model Suitability Assessment covers one model and configuration against agreed workload examples, with written results, failure cases, suitability findings and a findings call. We agree scope and data handling first. The aim is a supported decision about your actual work, before a convincing demonstration becomes an expensive deployment.