The Setup
Eight H100 SXM5 GPUs, a batched LLM inference service running a 70B-parameter model at fp16. The team was seeing GPU utilization hovering around 60-65% in steady state when they expected closer to 85%. The billing numbers matched: they were spending for peak capacity and getting mid-tier throughput.
They had a sampling profiler running. It showed the expected suspects in the top three: aten::linear, aten::scaled_dot_product_attention, and some attention mask work. Nothing surprising. Nothing that explained a 20-point utilization gap.
We attached zymtrace to the inference worker processes on two of the eight nodes and let it run at 99Hz for four hours during peak batch load. The results were not what any of us expected.
What the Continuous Profile Found
The flamegraph showed a wide orange band at roughly the same depth across every request path, sitting between the attention computation and the next linear layer. We followed the call path down: it was cudaMemcpyAsync calls, large ones, triggered by the KV-cache eviction logic that runs between attention heads when the sequence batch exceeds a certain length threshold.
The memory copies were host-to-device. The cache management code was moving KV tensors back to host memory and re-loading them in a pattern that made sense on paper (reduce GPU memory pressure for long sequences) but had never been measured for actual time cost at this batch size. At 99Hz continuous sampling, every one of those copies showed up in the profile. At 100Hz periodic sampling, the sub-15ms operations were statistically invisible.
Total measured time in those memory operations: 22% of GPU wall time across the full four-hour capture window. That was the utilization gap.
Why the Sampling Profiler Missed It
Sampling profilers work by capturing a stack snapshot every N milliseconds. At 100Hz, one sample lands every 10ms. A cudaMemcpyAsync call that takes 8ms will be captured in roughly 80% of sampling windows if it is the dominant operation. But these KV-cache copies were triggered per-sequence, per-batch, and they lasted 2-7ms each. At 100Hz, the probability of a single sample landing inside a 3ms window is 30%. Across 50 copies in a request, some will be caught, most will not.
The sampling profiler's histogram showed cudaMemcpyAsync at 3-4% of total samples, which looked minor. The continuous profile showed it at 22% of wall time. That difference is not measurement error; it is a fundamental artifact of the sampling approach applied to short, repetitive operations.
We have a separate post covering the sampling-vs-continuous math in more detail. The short version: sampling gives you a statistically correct picture of what runs longest in aggregate. It gives you a biased picture of what runs often and briefly. KV-cache management on H100 at high batch sizes falls in the second category.
Digging Into the H100-Specific Pattern
H100 SXM5 with NVLink 4.0 has 900 GB/s of GPU-to-GPU bandwidth and 80GB HBM3 per chip. The theoretical peak memory bandwidth for on-device operations is far higher than the PCIe lane bandwidth for any host-device transfer. So even a "small" host-device copy is expensive relative to what the GPU can do with on-device compute.
In this workload, the KV-cache management code had been written originally for A100 with 40GB HBM2. On A100, the threshold at which host offloading became cheaper than on-device compute was lower, because of the tighter HBM budget. On H100 with 80GB, sequences that triggered host offload on A100 should stay on-device entirely. But nobody had updated the cache policy thresholds when the service migrated to H100.
The flamegraph diff between the A100 and H100 profiles made the regression obvious. The H100 profile had a thicker orange band at the memory-copy depth. The A100 profile had a thinner one. Same code, different card, different cost profile.
What Fixing It Looked Like
The fix was three lines: raise the on-device threshold for KV-cache retention on H100, reducing the frequency of host offloads by approximately 70% under typical load. The work was not in writing the fix. It was in finding and understanding the problem before writing anything.
After the change, GPU utilization came up to 81-83% in steady state. Not the theoretical peak, but close enough that the remaining gap is explainable by attention computation padding on short sequences, which is a separate and understood problem with its own flamegraph signature.
The timeline from attaching zymtrace to having a confirmed hypothesis was about 90 minutes. Most of that was reading the flamegraph and cross-referencing the KV-cache code. The continuous profile gave us a clear call path down to the exact function issuing the copies, which ruled out a dozen other possible causes immediately.
What We Are Not Saying
We are not saying sampling profilers are useless. For finding the dominant hot path in a compute-bound workload where kernels run for hundreds of milliseconds, sampling is fast, cheap, and usually sufficient. The problem is specifically with short, high-frequency operations in mixed workloads where compute and data movement are interleaved.
We are also not saying every inference team has a 22% waste hiding in their memory copy paths. This workload had a specific trigger: the combination of long sequences, aggressive KV-cache management, and hardware migration without recalibrating operational parameters. But if you have moved a workload from A100 to H100 in the past year and have not re-profiled at the continuous level, it is worth a four-hour capture to find out what changed.
Running This on Your Own Cluster
If you want to reproduce this kind of profiling against a live inference service, the attachment is non-invasive:
pip install zymtrace
zymtrace attach --pid $(pgrep -n python) --duration 3600 --output profile.zmtFor multi-node clusters, run the same command on two or three nodes independently and compare the resulting flamegraphs side by side. Look for unexpectedly wide bands at any depth that do not correspond to the main compute kernels. Those are your candidates. The profile file loads directly in the zymtrace web UI or can be queried via the API for programmatic comparison.