How it works

zymtrace installs a lightweight agent binary on each GPU node. The agent uses eBPF to attach to Python and CUDA driver entry points without modifying your code or adding instrumentation. Sampling happens at the kernel level, which means it is invisible to the Python runtime and does not affect GIL behavior or CUDA stream scheduling.

The call-path graph (Python frames + CUDA kernel dispatches) is aggregated locally and shipped to the zymtrace backend in batches. You see a live flamegraph in the dashboard within seconds of the agent starting.

Architecture overview

  Your GPU Node
  ─────────────────────────────────────────────────
  [PyTorch model code]
      ↓ Python frames (eBPF uprobe)
  [Python interpreter]
      ↓
  [CUDA Driver: cuLaunchKernel / cuMemcpyAsync]
      ↓ kernel timing (eBPF kprobe)
  [GPU: H100 / A100]

  zymtrace-agent
  ├── sampler (99Hz, eBPF)
  ├── correlator (Python frame → CUDA dispatch ID)
  └── aggregator → HTTPS POST → zymtrace backend

Key concepts

Call-path correlation

The fundamental capability that separates zymtrace from GPU-only profilers: every CUDA kernel dispatch is tagged with the Python call stack that triggered it. This means your flamegraph shows model_forward / attention / F.scaled_dot_product_attention / cudnn_flash_attn_kernel rather than a list of GPU kernel names with no Python context.

Continuous vs on-demand

Most profiling tools require you to trigger a profiling session, run a specific input, collect results, and stop. zymtrace runs continuously. This matters because the expensive kernel launch patterns that inflate your p99 latency often do not appear in a controlled single-run profiling session. They appear under specific batch shapes, under memory pressure, or after many hours of operation. Continuous profiling catches all of them.

Zero-instrumentation

No decorators. No torch.profiler context managers. No changes to your model code. The eBPF agent attaches to the running process from outside using kernel uprobes and kprobes. Attach and detach are hot-path operations that do not require restarting your inference server.

Requirements

Next steps