The Problem With Small Kernels
Every CUDA kernel launch carries a fixed overhead on the CPU side: roughly 5-15 microseconds per launch for scheduling, argument marshaling, and driver interaction, depending on CUDA version, driver state, and whether the GPU is already warm. That overhead is negligible when your kernels run for milliseconds. It becomes the dominant cost when your kernels run for tens of microseconds each.
In LLM inference at small batch sizes (batch=1 to batch=4), this is a real problem. A single forward pass through a 7B-parameter transformer model at batch=1 launches roughly 3,000 to 5,000 CUDA kernels, many of which complete in under 50 microseconds. If you are targeting 50ms end-to-end latency, and each kernel launch costs 10 microseconds of CPU time, the launch overhead alone is 30-50ms of your budget before the GPU has done a nanosecond of useful work.
How Launch Overhead Accumulates
The overhead is not just raw launch time. The bigger problem is serialization. CUDA kernels submit to a queue (the CUDA stream), and the CPU must stay ahead of the GPU to keep that queue full. When kernels are short and numerous, the CPU can fall behind filling the queue, causing the GPU to stall waiting for the next kernel submission. You see this as GPU idle time between kernels in a CUDA timeline, which looks like low utilization even when the workload is theoretically compute-bound.
The timeline in NVIDIA Nsight Systems makes this visible: you can see gaps between kernel executions on the GPU timeline. But Nsight does not tell you which Python-level operation triggered the sequence of kernel launches that caused the gap. It shows you a flat list of kernels by name (entries like volta_s884gemm_fp16_128x128_ldg8_f2f_stages_32x5_tn) sorted by time. You know the gap exists; you do not know where in your model code to look for the fix.
What Call-Path Attribution Adds
zymtrace stitches the CPU call path to GPU kernel executions using eBPF hooks at the CUDA driver level. When you look at the flamegraph for a workload with high kernel-launch overhead, you see the launch sequence attributed back up the stack: torch._C._nn.linear calls to aten::addmm, which dispatches to the cuBLAS handle, which issues N individual kernel launches for the attention heads.
The specific pattern we found when running this against a small-batch inference server: one module, the rotary position embedding (RoPE) layer, was issuing 47 separate aten::mul calls per forward pass where 3 fused calls would have been sufficient. Each aten::mul translates to 1-2 kernel launches. At batch=1 and 40 requests per second, that is roughly 75,000 extra kernel launches per second, adding approximately 750ms of pure launch overhead per second of wall time. These are illustrative numbers derived from profiling a synthetic workload, not a specific deployment.
Finding the Pattern in a Profile
The flamegraph signature for kernel launch overhead is distinctive once you know what to look for. You see a wide flat bar at the cudaLaunchKernel depth that is not proportional to the kernel execution time shown on the GPU timeline. The CPU is spending time in launch bookkeeping that does not correspond to useful GPU work.
In the continuous profile, this shows up as a region where the flamegraph width (representing wall time) is dominated by CPU-side launch activity while GPU compute utilization is low. It is the inverse of the pattern you see in memory-copy-bound workloads, where GPU wall time is high but compute utilization is low for a different reason. The two patterns look similar in throughput metrics but have entirely different root causes and different fixes.
The fix is usually one of three things: fusing the redundant kernels via torch.compile or torch.jit.script to let the compiler merge them, reducing model operator granularity by rewriting the module to use fewer larger operations, or switching to CUDA graphs for the latency-sensitive path, which captures the kernel sequence once and replays it without per-launch CPU overhead.
CUDA Graphs as the Structural Fix
CUDA graphs deserve a note here because they directly address launch overhead. A CUDA graph captures the entire sequence of kernel launches and memory operations into a static graph object, then replays it with a single CPU call. The CPU overhead for replaying a graph is roughly constant regardless of how many kernels are inside it.
The tradeoff is that CUDA graphs require fixed tensor shapes and no dynamic Python control flow inside the captured region. For most decoder-only transformer inference at fixed sequence length, this is acceptable. For encoder-decoder models with variable output lengths, it requires padding or bucketing strategies to maintain fixed shapes.
We do not recommend CUDA graphs as the first intervention. Use them after you have identified the specific module responsible for high launch counts, verified that the kernel sequence is stable across calls, and confirmed the overhead is large enough to justify the added complexity. The flamegraph makes all three verification steps fast.
Kernel Launch Overhead at Scale vs. Single-Node
The pattern changes at scale. On a single inference node, kernel launch overhead is a CPU-side problem. On a multi-node setup with tensor parallelism and NVLink, the picture is different: ncclAllReduce calls between attention layer shards introduce synchronization points, and any imbalance in how quickly individual nodes reach those synchronization points compounds the launch overhead into collective stall time. One slow node's kernel launch queue backs up, delaying the synchronization, which stalls all other nodes.
In multi-node profiles, look for wide bands at the NCCL operation depth that are wider than expected for the data volume being communicated. Those bands often contain a mix of communication time and synchronization wait time. The call-path attribution shows you which layer's compute is running slow on the slow node, causing the stall.
Where Nsight Stops and Call-Path Profiling Starts
NVIDIA Nsight Systems is excellent for answering "how long did each kernel run?" and "what was the GPU timeline?" zymtrace answers a different question: "which Python function and which module produced this pattern of kernel launches?" Neither replaces the other. Nsight is the right tool when you already know which CUDA operation to investigate and want nanosecond precision on its execution. Call-path profiling is the right tool when you are looking for which part of a large model is generating unexpected overhead and need to trace back to the source code that produced it.
We are not saying Nsight is insufficient for its intended purpose. For CUDA kernel optimization work, writing custom kernels, tuning memory access patterns, optimizing warp occupancy, Nsight's kernel-level granularity is essential. The gap we are addressing is at the system level: understanding a large PyTorch model whose GPU cost you cannot attribute to specific modules without call-path continuity from Python down to the driver.