When an assistant feels slow, buying a faster GPU can seem the obvious answer. It may help. But if most of the delay is document retrieval, a queue or repeated model loading, a new GPU may leave much of the wait intact.

Start by recording where the time goes. A few measurements can narrow the investigation and give you a baseline against which to judge changes.

Northloom develops Krasis, a runtime for large-model inference on constrained GPU hardware, and offers inference performance assessments. That work involves separating prompt processing, generation, memory use and transfers; the same distinctions help diagnose a slow deployment. See our inference engineering work.

Separate the stages of a request

A typical application prepares the input, waits for inference capacity, processes the prompt and then generates a response. It may also validate the result or make further model calls before showing anything to the user.

For diagnosis, distinguish:

  • Input preparation: file reading, OCR, retrieval and prompt construction.
  • Queue time: waiting for the server to begin the request.
  • Prefill: processing the input tokens before generation.
  • Decode: generating the output tokens.
  • Application completion: validation, tool calls or other work needed to finish the task.

Measure time to first token and total completion time from the user’s side as well as any timings the server reports. Client time to first token can include network and queue delays. A runtime’s prefill measurement may exclude them. Check the definitions before comparing figures; serving systems such as vLLM document their queue, latency and token metrics.

Establish a repeatable baseline

Record the model revision, weight format, runtime version, hardware, input and output token counts, generation settings and concurrency. Note whether the model was already loaded and whether the request benefited from a prompt cache.

Run a short input and a representative long input, initially with one request at a time. Then repeat under expected demand. Use several runs so a single unusually fast or slow response does not decide the result.

Keep first-use results separate from warm results. Keep cache hits separate from fresh inputs. Both can matter operationally, but mixing them makes it hard to see what changed.

If the first response takes too long

Check when the inference server receives the request. If the application spends most of its time preparing documents or retrieving context, investigate that stage first.

If the request reaches the server promptly but waits there, look at queue depth, competing requests and capacity. A server that answers one person comfortably may become unresponsive during a burst.

If prompt processing dominates, compare short and long inputs. Applications can quietly accumulate large conversation histories or attach more documents than the task needs. Measure the actual token count reaching the model. Reducing unnecessary input may help, provided the shorter input still produces acceptable results.

Prompt caching can benefit repeated context when the runtime supports it and the requests qualify for reuse. Test with the mixture of new and repeated material your application actually sends. The TGI documentation on chunking and caching describes how these serving choices can affect request handling.

If the words arrive slowly

First confirm that the client receives streamed output as it is generated. An application or proxy that buffers a response can make a fast server appear unresponsive.

Then inspect where the model is running. Check GPU and CPU activity, memory use, transfers between devices and any competing processes. If weights or cached state move between system memory and GPU memory, those transfers may contribute to the delay.

The limiting factor depends on the model, runtime and workload. A low GPU utilisation reading is a clue to investigate, rather than proof that the GPU is unnecessary. Likewise, a high reading does not show that every operation is efficient.

Check support for the exact model and weight format in the selected backend. A configuration that loads successfully may use a slower execution path than expected. Compare supported configurations using the same workload, and check output quality after changes to precision or execution settings.

If it slows down with several users

Measure per-request latency alongside total throughput. A server may finish more work overall while each person waits longer. Batching and scheduling involve trade-offs between capacity and responsiveness.

Long requests can also change the experience of short requests. Test a realistic mix of input lengths and arrival times, including a busy period. If you have enough requests, inspect slower-percentile latency as well as the average; otherwise, keep the individual timings and describe the sample honestly.

Reduce concurrency as a diagnostic experiment. If latency improves sharply, you have evidence of a capacity or scheduling issue to investigate. It does not yet tell you whether the best response is different settings, another model or additional hardware.

Change one thing and check the result

Use the same baseline to compare each change. Record latency, completed work, peak memory, failures and output quality. A faster setting that causes missing fields, truncated responses or unacceptable answers has not met the original requirement.

Prefer reversible experiments: remove unnecessary context, correct an unintended CPU path, adjust a supported setting or isolate another workload. Keep a way back to the previous configuration.

If more hardware is needed, the measurements should explain why. They should also give you a workload and acceptance target to put in front of a supplier.

For an independent investigation, Northloom’s inference performance and optimisation assessment establishes an agreed baseline, investigates bottlenecks and reports prioritised recommendations. The measurements determine which changes are worth testing and whether they help.