A supplier can tell you how much GPU memory a server has. That does not tell you whether it can process your documents, support your staff or deliver a useful answer before they give up waiting.
Before asking for a hardware specification, write down the work you expect the system to do. A modest machine handling overnight document extraction has a different job from a shared assistant responding to staff throughout the day. Both may be described as an “AI server”, but they need different assessments.
Northloom develops the Krasis inference runtime and offers private AI infrastructure assessments. Our engineering approach starts with the model and workload, then tests whether the proposed hardware can meet them. See our inference engineering work.
Describe one real workload
Choose the first workflow you intend to deploy. Identify what goes in, what should come out and how you will judge the result.
For example, you might want to extract dates, parties and renewal terms from incoming contracts. That gives you something specific to test: representative contracts, an agreed output format and checked answers. “Run a capable local model” leaves too many important questions unanswered.
Record:
- The typical input and the largest input you need to support, measured in the intended model’s tokens as well as pages or file size.
- The expected output length and format.
- The number of requests arriving together during a busy period.
- How long a user can wait, or when a batch job must finish.
- The errors that require human review and the errors that make the workflow unacceptable.
Staff count is a poor substitute for simultaneous demand. A firm with 30 employees might normally have two people using the model at once, then ten during a training session. Test the pattern you expect to support.
Check model suitability before sizing the server
A fast model producing unreliable results is still an unsuitable system. Test a candidate model on your examples before making its hardware requirements the basis of a purchase.
Use the configuration you intend to run, including its quantisation, prompt template and document-processing steps. A result from a different model variant or a higher-precision configuration does not establish that your deployment will behave the same way.
Our guide to testing a local model on company workloads explains how to organise this evaluation.
Ask what the memory figure includes
Model files are only part of the memory requirement. The runtime also needs working memory, and many models keep an attention cache that grows as requests become longer. Concurrent requests can increase the requirement further. Cache behaviour varies by model and runtime; the Transformers cache documentation explains several of these trade-offs.
Ask the supplier to demonstrate the complete deployment at your intended input lengths and concurrency. Find out how much memory remains available during the busiest part of the test, rather than relying on the idle figure after loading.
A useful starting estimate for the raw weight storage of a dense model is parameter count multiplied by bits per weight, divided by eight. It is only an estimate: metadata, mixed precision, caches and runtime allocations still need space. For mixture-of-experts models, the parameters used for each token and the total weights that must be stored are different quantities.
If the proposal uses system memory or streams weights to the GPU, ask for a measurement of that arrangement. Being able to load a model establishes capacity; a workload test establishes whether it is fast enough.
Request a demonstration you can reproduce
Ask for the exact model revision, runtime version, settings, hardware and test inputs behind any performance claim. Then request results for:
- A normal request with one user.
- A long request with one user.
- Your expected busy period, including a mixture of request sizes.
- First use after startup, followed by repeated use with the model already loaded.
Measure the wait before the first response, the time to finish and the quality of the output. A single tokens-per-second figure cannot describe all three.
Agree whether prompt caching is enabled. Repeatedly sending the same document can give an optimistic impression if real requests usually contain new material. Include fresh inputs as well as any repeated context your application genuinely uses.
Include the cost of operating it
The purchase price is only one line in the budget. Establish who will install updates, investigate failures, manage access, monitor capacity and restore the service when something goes wrong.
Check power, cooling, noise, physical space, warranty, remote management and expansion options. Include software and model licences, support and staff time in the comparison. If you compare local operation with an external API, use the same workload volume, quality requirement and operating period.
Also trace the whole data path. A local model does not keep a workflow private if document conversion, embeddings or application logging send its contents elsewhere.
What to take into the buying decision
You should be able to connect the proposed hardware to an evaluated model, a defined workload and measured service targets. Any assumptions that remain untested should be written down, along with the cost and consequence if they prove wrong.
If a supplier cannot test your workload, consider a loan unit or a short rental evaluation before committing. A bounded assessment is easier to budget for than discovering after delivery that the server meets its specification but misses your needs.
Northloom’s infrastructure planning assessment reviews a proposed workload and deployment before you buy. If you already have a supplier specification, an architecture review provides a second opinion on its assumptions and trade-offs.