CPU profilers and GPU profilers answer different questions. This is not a shortcoming of either tool, it is a consequence of what each layer of the stack actually does. The problem for AI inference workloads is that the important costs are distributed across both layers, and treating them as separate views means you can optimize one half while the other continues burning compute and budget.
This post covers what each profiler type actually tells you, where each has blind spots for inference workloads specifically, and how a combined whole-stack view changes the picture.
What a CPU Profiler Tells You
A CPU profiler, whether it is a sampling profiler like py-spy or perf, or an instrumented profiler like cProfile, observes what the CPU is executing. For a Python inference service, this means you see the Python call stack: which handler function received the request, which tokenizer method ran, which model's forward() was called, and how the Python execution time distributes across those functions.
CPU profilers are accurate and low-overhead for what they measure. Sampling profilers add less than 1% overhead at typical rates. The output is easy to interpret for engineers familiar with Python performance work.
The fundamental limitation for GPU inference: most of the actual compute work happens asynchronously on the GPU. When your Python code calls torch.matmul(a, b) on CUDA tensors, the call returns almost immediately on the CPU side. The GPU executes the matrix multiply independently, while the CPU moves on to the next Python statement. A CPU profiler sees torch.matmul as taking 2 to 5 microseconds, which is the time spent dispatching the kernel to the GPU queue. The actual 4 milliseconds of matrix multiply execution on the GPU is invisible to the CPU profiler because the CPU was doing other things during that time.
The result: CPU profiler output for GPU inference workloads often looks deceptively cheap. Your model's forward() method might show 15 milliseconds of Python time while the actual GPU execution takes 80 milliseconds. The CPU profiler is accurate, it just does not see the GPU side.
What a GPU Profiler Tells You
GPU profilers observe device-side execution: which kernels ran, in what order, for how long, and with what hardware utilization. NVIDIA Nsight and CUPTI-based tools record precise kernel execution times, SM occupancy, memory bandwidth utilization, and other hardware counters.
For understanding whether a kernel is efficient given the hardware, GPU profilers are the right tool. If you want to know whether a particular matrix multiply kernel achieves good memory bandwidth utilization, or whether your attention implementation has warp divergence, you need GPU-side hardware counters.
The limitation for inference debugging: GPU profilers show you kernel names and execution times, but the kernel names in CUDA are often low-readability symbols like void at::native::vectorized_elementwise_kernel or ampere_fp16_s16816gemm_fp16_128x128_ldg8_f2f_stages_32x1_nn. Mapping these back to user code requires additional steps. Nsight provides some attribution when you instrument your code with NVTX ranges, but that requires code changes and does not help retroactively on a production service.
More practically: Nsight is designed for profiling runs, not continuous production observation. You cannot run a full Nsight profiling session on a live serving process without significant performance impact and operational disruption.
The Gap Between the Two Views
The most common scenario where the gap causes real problems: a team notices that end-to-end request latency is higher than expected but GPU utilization metrics look healthy. CPU profiler says Python execution is fast. GPU profiler (when run) shows kernels are individually efficient. Neither tool shows the problem.
The problem is usually in the transition between layers: implicit synchronizations, memory copies, or kernel launch overhead accumulating across many small operations. These are events that happen on the CPU side (synchronization points that block Python threads) but their cost is measured in terms of GPU idle time. A CPU profiler misses the GPU idle. A GPU profiler shows the idle but cannot tell you which Python call caused it.
Another gap: serial versus concurrent kernel execution. GPU hardware can run multiple kernels concurrently if they are issued on different streams and there is available parallelism. Code that could benefit from concurrent execution but is unnecessarily serialized on a single default stream has a cost that neither profiler type surfaces well. The CPU profiler sees calls completing at the expected rate. The GPU profiler shows kernels as sequential when they could be parallel. Without seeing both layers together you cannot identify the stream management improvement that would increase throughput.
How a Combined View Changes the Analysis
A combined whole-stack view, where CPU call stacks and GPU events are correlated on a shared timeline, makes the transition points visible. When a Python call causes an implicit synchronization, you see it as a wide stall frame in the combined stack, attributed to the specific Python function that triggered it. When memory copies serialize with kernel execution, you see both events in their call-path context.
The practical change is in what you investigate first. With separate CPU and GPU profiler outputs, engineers tend to focus on whichever tool shows something obviously expensive. With a combined view, the relative cost of different operations is visible in one place, and you can see whether the bottleneck is in Python orchestration, in the dispatch/synchronization layer, or in actual GPU kernel execution.
For most production inference services we have instrumented, the bottleneck is not in the kernel execution itself. Well-established operators like matrix multiply and attention kernels are already highly optimized by NVIDIA. The common sources of unexpected overhead are: unnecessary synchronization points, memory copies that could be eliminated or made asynchronous, launch overhead from many small kernels that could be fused, and Python-side overhead from operations that should be batched or vectorized.
When You Actually Need GPU Hardware Counters
We should be clear about where Nsight-style hardware profiling remains necessary. If you are writing custom CUDA kernels or evaluating the efficiency of a specific kernel implementation, hardware counters like memory bandwidth achieved versus theoretical maximum, SM occupancy, and warp stall reasons are the right metrics. A combined call-stack profiler does not replace hardware-counter analysis for this use case.
The workflow we recommend is layered: start with continuous whole-stack profiling to identify which operations are expensive in terms of attribution. Then, for the specific kernels identified as bottlenecks, use targeted Nsight profiling to understand hardware efficiency. This avoids the Nsight overhead on the full serving process while still allowing precise hardware analysis for specific operations.
A Note on Overhead
Continuous profiling has overhead. For CPU-side sampling at 99Hz with eBPF-based stack collection, the overhead is typically under 2% of CPU time and has no impact on GPU execution. CUDA event insertion for kernel timing adds a small constant cost per kernel launch, in the range of 2 to 8 microseconds per launch. For workloads with thousands of kernels per second, that adds up, which is why zymtrace's kernel timing mode is configurable and can be scoped to specific processes or call paths rather than run globally.
The value equation: 2% CPU overhead to make GPU attribution continuously visible in production is a reasonable trade for most teams. The alternative is flying blind until something goes wrong and then scrambling to profile a non-representative workload in a debugging session.