A GPU at 100% can be a sign of saturation, not success. In an AI application running on Kubernetes, the cluster may appear healthy while users wait too long for a response, or receive a quick but incorrect one. The challenge is connecting the user experience to the model, pods, and compute, storage, and network resources.
That is the problem The New Stack raises in an article about inference and observability, with contributions from Marius Bogoevici, Mahesh Ramachandran, and Karthik Subramanian of HPE. Their central argument is practical: accelerator activity alone does not show whether the service meets the business’s needs. It must be measured before production, and actual behavior must then be compared against that baseline.
“A quick first token can mask slow subsequent generation; a fast complete response can still be inaccurate or unhelpful.”
— Marius Bogoevici, Senior Principal Product Manager for HPE Morpheus Software
The First Response Does Not Tell the Whole Story
In a conversational service, time to first token (TTFT) measures how long it takes for the beginning of the response to appear. Time to complete the request, by contrast, includes generation of the rest. Between the two, inter-token latency—also called output token latency—helps track the generation rate once the model has started responding.
NVIDIA defines TTFT as the time from submitting a query to receiving the first token. That interval can include queueing, initial prompt processing, and network latency. End-to-end latency measures the time until the complete response is received and includes, among other components, queueing, batching, and the network. These are distinct measurements, not interchangeable variations of the same figure.
Input length matters too. NVIDIA notes that a longer prompt can increase TTFT because the model must process the input sequence before it begins generating. A measurement that observes only the first token can miss slow subsequent generation; one that calculates only total time can hide whether the user had to wait too long before seeing any result.
That is why Bogoevici recommends combining latency measurements with reliability, capacity, resource efficiency, cost, and response quality. Not all applications need the same targets: the right combination depends on the experience the team intends to provide. The criterion should not be an isolated number, but whether the service meets the objectives set for its specific use.
A Baseline Before Real Traffic
Mahesh Ramachandran recommends profiling the inference system before customers depend on it. Tests should reproduce representative prompt and context lengths, output volumes, and traffic levels. Under those conditions, teams can record latency, tokens processed, and service metrics, then use the results as a baseline for detecting changes in production.
Comparison is valuable when performed under sustained load and during demand spikes, not just in a brief test. It is also worth clarifying where each measurement starts and ends: if one tool calculates inter-token time differently from another, comparing their results can be misleading. NVIDIA warns that implementations vary and that definitions must match before figures are compared.
Aggregate performance is not the same as the individual experience, either. NVIDIA distinguishes between system tokens per second and tokens per second per user: as concurrency increases, total throughput may rise while the rate perceived by each person gets worse. The system can produce more overall while delivering a slower response to each request.
An NVIDIA methodology for NIM with Run:ai and Kubernetes without that scheduler illustrates how to build comparable tests: it evaluates workloads with Meta Llama 3.1 8B Instruct and DeepSeek R1 Distill Llama 8B, uses 2,000-token prompts and 200-token outputs for the summarization case, and scales pods and GPU under different configurations. The document also sets a latency threshold of around 1,000 ms to evaluate concurrent-user capacity. This is a methodology for those models and conditions, not a guarantee applicable to every service.
Scaling Does Not Always Eliminate the Bottleneck
High GPU utilization does not identify what is delaying requests. HPE suggests checking whether inference pods scale using metrics appropriate for the workload, whether requests reach the intended model, and whether related components can be scheduled when needed. A change in the mix of models can also affect latency and capacity, even if GPU activity appears normal.
The investigation can extend to network configuration, the response of a database or vector database, and the size of the GPU partition assigned to the model. These are possible areas to inspect, not universal explanations. The way to distinguish among them is to correlate affected requests and their time window with service and infrastructure signals.
Kubernetes allows the Horizontal Pod Autoscaler (HPA) to adjust the number of pods in a Deployment or StatefulSet using observed metrics. In addition to CPU and memory metrics, it can use custom or external metrics. So it is not necessary to replace the HPA just because CPU usage does not represent an inference workload well; the metric that reflects that workload does need to be provided and tested.
The HPA operates through a periodic control loop, not as a continuous reaction. Kubernetes documents a default 15-second interval between metric queries. Even if the signal indicates that more replicas are needed, scaling does not make GPU capacity available immediately. Configured limits, accelerator capacity, and the time required for pods to become ready all constrain what the policy can achieve.
There is also a tension between reacting quickly and avoiding oscillations. Kubernetes retains scaling recommendations over a window and, by default, stabilizes scale-downs for five minutes to smooth sharp drops in replica counts. For teams adjusting capacity to variable workloads, this can reduce fluctuations, although it means capacity is not cut in response to every drop in demand immediately.
Fractional GPU allocation is another decision that must be tested against the specific workload. NVIDIA’s methodology compares deployments using full and fractional GPUs for its models and test scenarios. The document also describes using Run:ai to adjust replicas based on throughput, latency, or concurrent users. These results do not support concluding that a particular fraction will suffice for another model: the profile and memory requirements affect capacity per replica.
Observability, Quality, and Cost in the Same Decision
HPE presents Morpheus as a provisioning and automation platform, and OpsRamp as an infrastructure and application observability and monitoring tool. According to Ramachandran, the goal of combining them is to relate model and agent activity—such as token volumes and rates, inference latency, and TTFT—to supporting telemetry. The value lies in narrowing down whether a slow response points to the application, the model service, or the infrastructure; each deployment must verify which component emits each signal and what additional instrumentation it needs.
Technical observability does not determine whether the response is good, either. Bogoevici proposes evaluating whether the service responds correctly or hallucinates. Telemetry can confirm that a request was handled, but quality requires representative tests and acceptance criteria defined by data teams and domain specialists. That evaluation should be repeated after changes to the model, prompt, information retrieval, or serving system.
Measuring more also means deciding what data is recorded. Bogoevici warns that prompts, responses, and retrieved documents may appear in logs and traces. Therefore, teams need to define what is collected, where it is stored, for how long, and who can access it. In particular, sensitive material does not stop being sensitive just because it was captured to diagnose a problem.
The launch decision brings together issues often spread across several teams: performance, quality, capacity, cost, security, and failure response. Ramachandran calls for agreement on the expected business value, including a plausible inference volume tied to a business outcome. Subramanian argues that approval cannot rest with one person: platform administration, machine learning, data science, and security must be involved, along with the relevant business owner.
The operational implication is clear: an occupied GPU is not enough as a success criterion or as the sole trigger for scaling. The baseline must link user wait time and response quality to infrastructure capacity and status. After launch, comparing those measurements with pre-launch tests makes it possible to detect regressions and decide whether the model, request routing, or available resources need adjustment.
Sources: The New Stack, NVIDIA, NVIDIA, Kubernetes
