How Sampling Profilers Work and Why That Matters
A sampling profiler captures a stack snapshot at fixed intervals. At 100Hz, one sample lands every 10 milliseconds. If a function runs for 200ms, it will appear in approximately 20 samples, which is a statistically useful representation of its cost. The key word is "statistically useful": for long-running work, sampling gives you an accurate picture of relative cost between functions.
The problem emerges when your workload consists of many short operations. A CUDA kernel that takes 3ms will appear in roughly 30% of 10ms sampling windows. A kernel that takes 0.5ms will appear in 5% of windows. If that 0.5ms kernel runs 200 times per request, it contributes 100ms of total GPU time per request, which is a major cost center. But it will appear in only a handful of samples across the profiling window, making it look like a rounding error rather than a primary target.
The Specific Problem With Transformer Inference
Transformer inference does not look like a single long compute task. A 70B-parameter model at batch=1 runs thousands of kernel invocations per request: linear projections, attention computations, activation functions, layer norms, each lasting microseconds to low milliseconds. The request pipeline is a high-frequency mix of compute and data movement, not a monolithic long operation.
At 100Hz sampling, the math does not work out. Consider a typical A100 inference workload:
- Attention QKV projection kernel: 1.2ms, runs once per layer per request
- KV-cache copy on sequence overflow: 4ms, runs on 15% of requests
- Element-wise activation kernels: 0.3ms each, runs 3 times per layer per request
At 100Hz, the KV-cache copy (4ms) will be captured in about 40% of sampling windows where it runs. The activation kernels (0.3ms) will be captured in about 3% of windows. But the activation kernels run far more frequently. If the service handles 80 requests per second across 32 layers, those activation kernels run 7,680 times per second for a total contribution of 2.3ms per second of wall time. A sampling profiler will show them as minor cost items. They might be the primary bottleneck.
What Continuous Profiling Measures Instead
Continuous profiling, as zymtrace implements it, does not take statistical samples. It hooks into the CUDA driver at the eBPF level and records every kernel launch event: the kernel name, the calling stack at the time of dispatch, the CUDA stream, the device-side duration measured with CUDA events. Every kernel, every call.
The flamegraph built from this data is not a statistical estimate of relative cost. It is an accounting of actual wall time attributed to actual call paths. If the KV-cache copy costs 4ms and runs on 15% of requests, it shows up as 15% frequency contribution at that call depth, with the full Python call stack showing which function triggered the copy.
The important distinction: sampling profilers produce an approximation of cost distribution. Continuous profiling produces an exact record of what happened. For workloads with high-frequency short kernels, the approximation error in sampling is not evenly distributed. Short kernels are systematically underrepresented.
The 2ms Boundary
At 100Hz sampling, any kernel that takes less than 10ms is underrepresented to some degree. Below 2ms, you need to treat sampling data as directionally useful at best and potentially misleading for specific operations. Below 0.5ms, the sampling data about individual kernel costs is largely noise for those specific operations.
Why 2ms specifically? It is not a sharp boundary. It is a practical threshold where the expected number of samples per kernel invocation drops below 1, meaning individual kernel runs will not always appear in the profile at all. An operation that runs 1,000 times per second and takes 1.5ms will be captured in about 15% of its invocations at 100Hz. That 15% sample is enough to tell you the function exists and is somewhat hot, but it will rank it far below its true cost contribution in the sorted profile output.
Overhead: What Continuous Profiling Actually Costs
The objection we hear most often: continuous profiling sounds expensive. Recording every kernel launch event sounds like it would add meaningful overhead to inference throughput.
The overhead of zymtrace at 99Hz is measurable but small: below 1% of throughput impact on GPU-bound inference workloads in our testing. The eBPF hooks at the CUDA driver level are in the CPU path, not the GPU execution path. The GPU itself runs exactly the same kernel sequence it would run without the profiler attached. What changes is the CPU side: a small amount of per-call bookkeeping when the kernel is dispatched from the CPU.
For workloads that are CPU-bound (unusual for large model inference, but possible at very small batch sizes), the overhead can be higher because the CPU is already the constraint. In that case, sampling profiling is a better fit for production monitoring, and continuous profiling is better reserved for dedicated diagnostic captures on a staging system.
When Sampling Is Still the Right Tool
We are not arguing that sampling profilers should be replaced. Sampling is the right tool when you want a lightweight, always-on view of relative cost across a large application. A Python sampling profiler at 10-50Hz is an excellent first-pass diagnostic for finding which modules are doing the most work overall. For CPU-side bottlenecks in the data pipeline, in the tokenizer, in the batching logic, sampling gives you accurate relative cost at minimal overhead.
The specific gap is in GPU kernel attribution for high-frequency short-duration kernels. That is where sampling systematically misleads. Use sampling for system-level orientation and continuous profiling for kernel-level attribution, especially when investigating GPU utilization gaps that sampling cannot explain.
A Practical Test to Check Whether Your Profile Is Lying
Run both profilers against the same workload simultaneously. Compare the top-10 cost items by total attributed time. If the lists are broadly similar (same modules, roughly proportional ranking), sampling is giving you a good picture and continuous profiling is confirmatory. If there are items in the continuous profile's top-10 that are not in sampling's top-10, those are your sampling blind spots. In our experience with transformer inference workloads, one to three such items typically emerge, and at least one of them is a meaningful cost driver.