If you have used a sampling profiler before picking up a flamegraph tool, you probably have an instinct to look for the widest bar and treat it as "the thing to fix." That instinct works reasonably well for CPU profiling on single-threaded code. It does not transfer cleanly to GPU inference workloads, and acting on it can send you optimizing functions that have nothing to do with your actual bottleneck.
This post is for SREs and platform engineers who are new to reading flamegraphs in a GPU context. The goal is to explain the one conceptual difference that changes how you interpret the output, and then walk through several real reading patterns that come up when profiling inference workloads.
What Width Actually Means
In a flamegraph, the horizontal axis is cumulative CPU (or GPU) time spent in a call path, not wall clock time. A frame's width represents the total time the call stack was observed in that function and everything it called, across all samples in the profiling window.
In a flat histogram view (like the one cProfile outputs, or the sorted table in py-spy's default output), each row shows either total time (inclusive) or self-time (exclusive) for a function. The histogram answers "which function was sampled most often."
The flamegraph answers a different question: "what is the call path that led to this function being sampled." A function can appear wide in a histogram because it is called by many different callers. The flamegraph separates those callers into their own stacks, so you see not just that a function is expensive, but why it is being called and what called it.
This distinction matters because the fix for "this function is slow when called from path A" is often different from the fix for "this function is slow when called from path B." The histogram merges those two problems into one row. The flamegraph keeps them separate.
The Histogram Trap in Practice
Here is a concrete example pattern we encounter regularly. A team profiles their inference service and sees torch.nn.functional.layer_norm accounting for 8% of GPU time in the aggregate profiling data. They treat this as a high-priority optimization target and start researching fused layer norm kernels.
In the flamegraph view, layer_norm appears in two separate call paths. About 5% of its total time comes from the attention sublayer, which is expected and well-optimized. The other 3% comes from a preprocessing step that is calling layer norm on each token individually in a loop, rather than batching the operation. The loop is not visible in the histogram because it is spread across many small calls that each look cheap. The flamegraph shows the loop call path as a repeating narrow stack, and the total width of those stacks matches what the histogram attributed to layer norm.
Fixing the fused kernel implementation in the attention path would have had minimal impact because that code was already running close to hardware limits. The actual fix was a 3-line change in the preprocessing loop.
Reading GPU Flamegraphs: Layers From Top to Bottom
In zymtrace's flamegraph output for a GPU inference workload, the stack grows downward from the root frame. The layers you typically see are:
The topmost layers are Python frames: the request handler, the model's forward() method, the operator calls like torch.matmul or torch.nn.functional.linear.
Below the Python frames, you cross into the C extension layer: aten::mm, aten::addmm, or similar ATen operations. These are the kernel dispatch points where PyTorch selects which CUDA kernel to run.
Below ATen, you enter the CUDA driver and runtime layer: cudaLaunchKernel, cuLaunchKernel, and then the actual kernel symbol names. If a memory copy is involved, you will see cuMemcpyHtoDAsync or cudaMemcpy variants here instead of kernel launch calls.
The width of the Python frame includes all the time in everything below it. The width of the ATen frame is narrower because it excludes the Python dispatch overhead. The width of the CUDA kernel frame is the actual device execution time plus the driver-side launch cost.
Tall Stacks Versus Wide Stacks: What Each Signals
A tall, narrow stack in the flamegraph means a deep call chain where no single layer is consuming most of the time. This is common in dispatch-heavy code where PyTorch is doing a lot of bookkeeping to reach the actual kernel. Tall narrow stacks are sometimes actionable (you can shortcircuit dispatch with torch.compile or Triton kernels in specific cases), but they are rarely the primary bottleneck.
A wide, shallow stack means you are spending a lot of time in a small number of call paths with limited nesting. This is what hot paths look like. The width is proportional to the time, and the shallowness means the cost is concentrated rather than distributed across many layers.
A series of similar-height stacks repeated across the flamegraph indicates a loop: the same call path executed many times. This is the pattern to look for when you suspect per-token or per-request overhead that should be batched.
Concurrency and Why a GPU Flamegraph Is Not a CPU Flamegraph
CPU flamegraphs typically represent a single thread or a merged view of multiple threads. GPU flamegraphs have a structural difference: the GPU executes kernels from a queue, and multiple kernels can run concurrently on different SM partitions if they fit. A naive flamegraph that shows all GPU work as sequential would be misleading for understanding actual throughput.
zymtrace resolves this by attributing kernel execution time to the call path that dispatched it, not by drawing kernels as sequential time slices. This means the flamegraph represents attribution rather than a timeline. Two kernels that ran concurrently are both shown as full-width frames under their respective dispatch paths, not as half-width frames sharing the same time slot.
The practical implication: the sum of all frame widths in a GPU flamegraph will often exceed the actual wall clock duration of the profiling window. That is by design. What matters is the relative width of different paths, not whether the total adds up to 100% of elapsed time.
What to Look for First
When you open a flamegraph for the first time for an inference workload, start with these three questions.
First: where is the widest frame below your model's forward() call? That is your first candidate for investigation. Check whether its width matches your expectation based on what you know about the model architecture.
Second: are there memory copy frames (cudaMemcpy, cuMemcpyHtoD) at significant widths? Memory copies do not show up in GPU utilization metrics but they stall the compute pipeline. If they are wide, they deserve attention before any kernel-level optimization.
Third: do you see repeated stacks? A loop that should be batched will appear as a repeated pattern. Count the occurrences and multiply by the per-instance width to get the total cost. If the aggregate is significant, batching that operation is often the highest-value change you can make.
The histogram is still useful for a first pass at what functions consume the most time in aggregate. Use it to prioritize which call paths to investigate in the flamegraph, not as the final answer about what to change. The flamegraph is where you find the why behind the what, and the why is what determines whether a change is actually worth making.