The A100 has 2 TB/s of HBM2e memory bandwidth. That number appears in every vendor performance comparison, and it is accurate. What it does not tell you is that your process may be spending a significant fraction of GPU time waiting for data to arrive from host RAM, with kernel execution stalled behind it.
We looked at 40 production inference workloads over the past several months. In 28 of them, cudaMemcpy or one of its driver-level equivalents appeared in the top-5 cost paths when measured by wall-time on the GPU. Some cases were obvious. Most were not.
How Memory Copy Stalls Actually Work
The A100's compute engines and its memory DMA engine operate independently. A kernel running on the SMs and a DMA transfer in progress can theoretically overlap, provided they are issued on different CUDA streams and there is no dependency between them. The problem is that most production code does not actually achieve this overlap.
The most common failure mode: cudaMemcpy (the synchronous variant) forces the calling CPU thread to block until the transfer completes. CUDA's internal stream synchronization ensures no subsequent operations on the default stream start until that copy finishes. If your tokenizer runs on the CPU and you copy the token tensor to device before every forward pass, that copy serializes with the previous inference batch's kernel execution.
The transfer rate over PCIe 4.0 x16 peaks around 32 GB/s, but typical real-world throughput is 18 to 22 GB/s after protocol overhead. For a 512-token sequence with a 32K vocabulary, the input tensor is around 2 MB. At 22 GB/s that is roughly 90 microseconds of transfer time. Over 1,000 requests per second, that is 90 ms of wall time per second spent in memory transfer, not compute.
Why Standard Metrics Do Not Surface This
DCGM and standard NVIDIA metrics track GPU utilization at a polling interval of one second by default. A 90-microsecond stall per request does not register as a utilization drop at that granularity. nvidia-smi will show 95% GPU utilization while your process is serializing on memory copies that account for 9% of actual request latency.
CPU profilers like py-spy sample Python stacks. They will show you model.forward() taking the expected time, but they have no visibility into what is happening on the GPU side. The memory copy overhead is buried inside the CUDA API call and does not appear as Python execution time.
NVIDIA Nsight can surface this, but it requires an instrumented profiling run rather than continuous production observation. You also need to know what to look for before running it, or the output is 50,000 rows of kernel events with no obvious starting point.
What the Flamegraph Pattern Looks Like
In a zymtrace flamegraph, memory copy stalls show up as wide frames attributed to cuMemcpyHtoDAsync, cudaMemcpy, or cudaMemcpyAsync in the CUDA driver layer. The frame spans both the CPU-side blocking time and, when we can correlate it, the GPU-side DMA execution time.
The key tell: these frames sit directly below torch.Tensor.cuda() or similar host-to-device calls, and they appear at regular intervals corresponding to your batch cadence. When you see a wide memory-copy frame followed by a kernel execution frame of similar width, you are looking at sequential transfer and compute where the two could potentially overlap.
A different pattern appears when the stall comes from an implicit synchronization, like calling .item() on a tensor to extract a scalar. That pattern shows cudaDeviceSynchronize in the driver layer, with a forced device-to-host copy preceding it. These are often harder to spot because the triggering Python call looks innocuous in a CPU profiler.
Three Patterns We See Most Often
The first pattern is eager host-to-device transfer at the beginning of each forward pass. The tokenizer outputs a NumPy array or CPU tensor, and the code calls .to('cuda') or .cuda() synchronously before the batch is dispatched. The fix here is usually to pin host memory with pin_memory=True on the DataLoader or manually with tensor.pin_memory(), then use non-blocking transfers, handling synchronization explicitly with CUDA events.
The second pattern is KV cache reads from CPU memory during constrained-memory serving. Under memory pressure, some serving frameworks evict KV cache blocks to host RAM and restore them on demand. Each restore is a host-to-device copy that blocks the decode step. This is harder to fix because it is architectural rather than a simple API change. Making it visible in the flamegraph is the prerequisite for addressing it.
The third pattern involves loss or metric logging that extracts GPU values to the CPU inside a training or fine-tuning loop. A call like loss.item() or accuracy.cpu() inside the training loop forces a device-to-host copy and a device synchronization. If this happens every step, it can add 3 to 8% to total training time on large models, depending on model size and step frequency.
What to Fix and What to Accept
We are not saying synchronous memory copies are always wrong. There are cases where you genuinely need the data on the CPU before proceeding, and hiding the latency in a background stream just moves the stall later in the pipeline. The question is whether the copy and the subsequent kernel execution have a true data dependency or whether you are serializing them out of habit.
For inference serving, the highest-value change is usually pinned memory combined with non-blocking transfers and explicit prefetching. Structure your pipeline so the token tensor for batch N+1 is being transferred to device while batch N is executing. This requires CUDA streams and attention to synchronization points, but the latency improvement in high-throughput scenarios is direct and measurable.
For the KV cache eviction case, the fix is at the serving framework level. Knowing the pattern exists lets you prioritize the right architectural work: whether that is expanding device memory allocation, implementing smarter eviction policies, or accepting the cost for specific workloads. None of those decisions are possible if the stall is invisible in your observability stack.
The logging pattern is the easiest to fix: batch your metric extraction, or extract on a subset of steps rather than every iteration. Log a running average every 100 steps rather than calling .item() every step.
Making the Invisible Visible
The underlying problem with memory copy stalls is not that they are technically complex to fix. Most of them are addressable with known patterns. The problem is that they sit in a blind spot between CPU profilers and GPU utilization metrics. CPU profilers do not see GPU activity. Utilization counters smooth over microsecond-scale stalls. The result is that teams optimize the Python code, achieve 95% GPU utilization in aggregate metrics, and still leave 10 to 15% of actual compute throughput on the table.
Continuous whole-stack profiling surfaces these stalls as concrete call-path entries with duration attached. Once the stall has a name and a width in the flamegraph, it becomes an engineering problem rather than a monitoring gap. That is the shift worth making.