What Kernel Fusion Actually Is
When PyTorch evaluates a sequence of element-wise operations, say a multiply followed by a ReLU followed by a layer norm, the naive implementation issues one CUDA kernel per operation. Each kernel reads its input from GPU memory, performs its computation, writes its output back to GPU memory, and terminates. The next kernel then reads that output from memory. For a tensor of 1M elements at fp16, that is three memory round-trips for what could be accomplished in one pass if the operations were fused into a single kernel.
Kernel fusion collapses this sequence into one kernel that reads input once, performs all three operations in registers, and writes output once. Memory bandwidth is often the bottleneck on modern GPUs for element-wise operations: HBM bandwidth on H100 is high in absolute terms but is a shared resource across all kernel activity. A 3x reduction in memory traffic for a common operation sequence translates directly to throughput improvement and lower time-per-token in inference.
How PyTorch Decides to Fuse
torch.compile with the TorchInductor backend is the current mechanism for automatic fusion in PyTorch 2.x. It analyzes the computation graph and identifies fusable op sequences using pattern matching and loop fusion passes. For standard model architectures with static shapes, it works well. The compiled version of a simple MLP block can fuse 8-10 element-wise operations into 2-3 kernels and achieve measurable throughput improvement on the same hardware.
But fusion is not guaranteed. TorchInductor has a set of conditions under which it will fuse: the operations must be adjacent in the graph with no intervening data-dependent branches, must operate on compatible tensor layouts with no implicit transposes requiring a layout change between ops, and must have shapes within the fusion budget (very large tensors may be split across multiple kernels for tile size reasons). When any condition is violated, fusion falls back silently, and you get unfused execution with no warning.
The Failure Modes We See Most Often
The most common fusion failure in production inference code is the custom module antipattern: a developer writes a component using standard PyTorch ops, it works correctly, and nobody checks whether the resulting kernel sequence is actually fused. By the time the model is serving traffic, the unfused version is running correctly, so there is no error signal that prompts investigation.
Specifically, these patterns generate unfused sequences most frequently:
- In-place operations with aliased tensors.
tensor.add_(other)followed by another operation that readstensorbefore the add completes can prevent fusion if the compiler cannot prove the alias is safe. - Non-contiguous tensors from
.view()or.permute()without explicit.contiguous(). Layout changes break fusion boundaries in TorchInductor's current implementation. - Custom autograd functions. Any
torch.autograd.Functionsubclass is opaque to the compiler; fusion stops at its boundary. - Dynamic shapes without the
dynamic=Truecompile flag. A compiled model that receives inputs with unexpected shapes falls back to eager mode, losing all fusion.
What the Flamegraph Shows
In a profile of a model with good fusion, the kernel execution pattern is clustered: a few large kernel calls interleaved with the attention and linear layers. In a model with fusion failures, you see a staircase pattern: many small kernels at the same call-path depth, each lasting 10-50 microseconds, adding up to significant total time.
The zymtrace flamegraph shows this at two levels. The Python stack shows which module produced the kernel sequence, tracing the call depth from your module's forward method down to the aten:: operations. The GPU timeline shows kernel execution density. When a module produces a dense cluster of short GPU kernels, the combination of both views makes the unfused region obvious. You can compare the flamegraph of the compiled model against the eager model profile and see exactly at which depth fusion stopped working.
A Concrete Example: Attention Score Normalization
Consider a self-attention module that computes its own normalization inline: after computing attention scores, it applies a custom scaling and clamping operation written as three separate PyTorch ops. This pattern, common in older model implementations written before torch.compile was production-stable. In the compiled profile, the compiler may successfully fuse the first two ops but not the clamp, when the clamp threshold is stored in a Python variable rather than a tensor constant, which causes the compiler to insert a host-side synchronization point to read the threshold value.
The result is one fused kernel for the first two ops, a host sync, and then a second kernel for the clamp. The host sync introduces a CPU-GPU synchronization that serializes execution and costs approximately 15-25 microseconds per attention layer call. For a 32-layer model at 100 requests per second, that is between 48,000 and 80,000 synchronization stalls per second. The continuous flamegraph shows them as a thin but consistent orange band at the attention normalization depth, totaling 4-8% of end-to-end inference time in this pattern.
The fix: convert the Python-variable threshold to a registered buffer with self.register_buffer('scale_clamp', torch.tensor(threshold)), which makes it visible to the compiler as a tensor constant and allows full fusion. Three lines of change for a measurable throughput gain.
Why This Stays Hidden Without Continuous Profiling
The characteristic of fusion failures is that they are invisible to any tool that does not connect the Python call graph to the GPU kernel sequence. A sampling profiler shows you which Python functions consumed the most CPU time. GPU utilization metrics show you whether the GPU is busy. Neither tells you that a specific module is generating 10 short GPU kernels where 2 fused kernels would cover the same computation.
The continuous flamegraph connects those layers. Because it captures the CPU stack at each CUDA API call and records device-side kernel duration from CUDA event timing, it shows you both the Python source and the resulting GPU kernel pattern in a single view. The unfused region appears as a wide staircase at the module's depth in the flame; the fused region appears as a single wide bar. The visual contrast makes the regression findable without knowing in advance which module to suspect.
What We Are Not Claiming
We are not saying torch.compile is broken or that most models have serious fusion failures. For well-maintained model implementations using standard architectures, the compiler fusion rates are high and the gaps are small. The problem is concentrated in custom research code, domain-specific models with unusual op sequences, and models that were written before eager-mode profiling was part of the development workflow.
We are also not saying continuous profiling is the only way to find these issues. torch._dynamo.explain() can tell you why the compiler failed to fuse a specific graph region. The difference is that explain() requires you to already know which module to investigate. The flamegraph shows you which module to look at first; then explain() tells you the specific reason fusion failed. They work best together.