The Problem That Does Not Exist on a Single Node
On a single GPU node, continuous profiling is straightforward: attach an eBPF agent to the process, hook the CUDA driver, record kernel events and call stacks, write to a local ring buffer, flush to storage asynchronously. The overhead is bounded by the CPU cost of the hooks and the I/O cost of the flush. Neither grows with the number of GPU kernels executing in parallel on that node.
A multi-node inference cluster introduces three problems that single-node profiling does not have: how do you correlate profiles across nodes without synchronizing clocks at inference time, how do you avoid adding network traffic that competes with model communication (NCCL all-reduces on NVLink or InfiniBand), and how do you aggregate the per-node flamegraphs into a cluster-wide view without losing the per-node attribution needed to diagnose inter-node imbalance.
This post covers the architectural decisions we made building zymtrace's multi-node collection layer, and what the resulting flamegraph looks like when you are debugging a 16-node A100 cluster running tensor-parallel inference.
Clock Skew and Why You Cannot Ignore It
Every profiling timestamp is meaningless across nodes unless the clocks agree. NTP synchronization on modern Linux systems is typically within 1-10ms of accuracy. CUDA events have microsecond resolution. If you naively merge profiles from two nodes that disagree by 5ms, you will see phantom serialization in the flamegraph: operations that actually ran in parallel on both nodes appear to be sequential, because one node's timestamps are shifted relative to the other.
The standard approach for distributed tracing (used by Jaeger, Zipkin, and similar systems) is to propagate a trace context through the request so that events on different nodes can be correlated to the same logical request. For inference workloads with tensor parallelism, the request does not naturally flow through all nodes sequentially; instead, all nodes execute the same layer at the same time and exchange tensors via collective operations. The trace correlation has to happen at the collective operation boundary, not at the request entry point.
zymtrace hooks into the NCCL collective operations via eBPF and records a correlation ID at each collective call, using the NCCL communicator handle as a natural grouping key. This creates per-layer correlation points that are node-independent: even if node clocks disagree by a few milliseconds, the layer N all-reduce on all nodes maps to the same logical event. The clock skew between nodes can then be measured from the timestamp distribution of these correlated events and subtracted as a calibration offset before flamegraph rendering.
Not Adding Traffic to the Training Network
On a multi-node GPU cluster, the inter-node network (typically InfiniBand or NVLink fabric, sometimes both) carries both model weights at startup and NCCL collective operations during inference. Any profiling data export that competes with NCCL traffic on the same network path can increase the latency of collective operations, which stalls all nodes simultaneously.
The zymtrace agent uses the management network (separate NIC, typically 1GbE or 10GbE out-of-band) for profile data export, not the compute network. The agent buffers profile data locally in a ring buffer per node and flushes to the profiling backend over the management network in background batches. The flush is rate-limited and backpressure-controlled: if the management network is saturated (which is unusual, but possible during large flamegraph exports), the ring buffer ages out the oldest events rather than blocking kernel launch recording.
This means the profile data is slightly lossy under extreme flush load, but the loss is bounded and predictable. In practice, at 99Hz sampling on a 16-node cluster, the per-node data rate is roughly 2-5 MB/min depending on call stack depth. That is well within typical management network capacity even with all 16 nodes flushing simultaneously.
What the Cluster-Wide Flamegraph Looks Like
The cluster-wide view aggregates all node profiles aligned by layer index (using the NCCL collective correlation). The x-axis represents wall time, the y-axis represents call depth, and color intensity represents attribution (darker orange = more time spent at that path). For a balanced tensor-parallel workload, all nodes should show identical flamegraph shapes: same layer structure, same kernel durations, same collective widths.
When a node has an imbalance (slower compute, memory bandwidth saturation, a different kernel dispatching pattern due to a software state difference), it shows up as a wider bar at a specific layer depth on that node's lane compared to the others. The cluster-wide view makes this imbalance immediately visible without requiring you to compare 16 individual flamegraphs manually.
The most common imbalance we see in practice: one node is running a different version of the model weights (a partial update during a rolling restart) or has a different CUDA driver version that chooses a different cuBLAS kernel variant for a GEMM operation. Both show up as a persistent width difference at the linear layer depth for that node. The call path attribution distinguishes the two causes: weight mismatch shows up as slower data loading at the embedding depth; driver kernel selection shows up as different kernel names in the CUDA API hook layer.
The 99Hz Design Decision
We chose 99Hz rather than 100Hz for the continuous sampling rate for a specific reason: 100Hz aliases with many system timer frequencies (100Hz is exactly the CONFIG_HZ=100 Linux kernel tick rate on some configurations). When the profiler fires at exactly the same frequency as the kernel timer, samples cluster around tick boundaries and miss the activity between ticks. 99Hz creates a slight phase offset that rotates through the tick period over time, distributing samples more uniformly across the actual execution timeline.
The overhead difference between 99Hz and 100Hz is negligible. The sample distribution improvement is real, particularly for workloads with periodic structure (like inference loops with fixed request rates). At a 50-requests-per-second inference server, 100Hz sampling would produce a rhythmic aliasing with the request cadence (100 / 50 = 2 samples per request, perfectly synchronized). At 99Hz, samples rotate through the request period and capture different phases of each request over time.
Running Continuous Profiling in Production
The practical question: should you run continuous profiling in production or only in staging? Our answer is production, but with the ring buffer configured for a short retention window (5-10 minutes) under normal operation. The profiler is always running and capturing, but it only ships data to the backend when you explicitly trigger an export window.
This design gives you two benefits: when an anomaly occurs (a latency spike, a cost outlier, an unexpected GPU utilization drop), you can export the last 5 minutes of continuous profile data and have a full kernel-level record of exactly what happened, not a reconstructed profile from a post-hoc diagnostic session. And you avoid storing and processing full continuous profiles indefinitely, which would be costly at scale.
The trigger for export can be manual (a one-line CLI command), metric-driven (export when p99 latency exceeds a threshold), or alert-driven (triggered by your existing alerting system via the REST API). The architecture does not prescribe which trigger you use; it only ensures that when you need the data, it is already captured.
What This Does Not Solve
Cluster-wide continuous profiling does not replace application-level distributed tracing for latency attribution above the GPU layer. If your inference service has variable latency coming from load balancing, request queuing, tokenizer overhead, or network routing, those contributors live in the CPU and network layer above what GPU profiling captures. The zymtrace cluster profile tells you what happens inside the GPU execution graph once a request has reached the inference worker and dispatching has begun. For the end-to-end latency picture, you still need both.