Your whole AI stack. One flamegraph. Always on.
zymtrace runs as a lightweight agent alongside your GPU workloads. No code changes. No sampling gaps. The call path from your Python model code to the CUDA kernel that ran it, captured at 99Hz, stored and queryable.
Four capabilities working together
No single profiling tool covers the full path from Python orchestration to GPU kernel. zymtrace stitches all layers into one view.
Always sampling, never invasive
zymtrace runs a 99Hz profiling loop using Linux eBPF, attaching to both the Python interpreter and the CUDA driver without modifying a single line of your code. It captures frames continuously across every inference request.
- eBPF-based, zero code instrumentation required
- Runs 24/7 without throughput impact
- Captures rare events, not just averages
Python to CUDA in one call path
Most profilers show you Python or GPU, never both connected. zymtrace correlates Python frames with the CUDA kernels they trigger, so you see the complete path: which model layer called which kernel, and what that kernel cost.
- Python frame to CUDA kernel correlation
- Stream-aware GPU timeline visualization
- Works with PyTorch, JAX, and TensorFlow
Flamegraphs that tell you the actual cost
A flat histogram hides the call path. A standard flamegraph hides concurrency. The zymtrace UI shows you exactly which call chain is responsible for which GPU time, with attribution to specific kernels, not averages.
- Click to drill down to any frame
- Compare before and after deploys
- Time-range selection over live data
Tie GPU spend to model components
Once you know which CUDA kernel costs what, zymtrace traces that cost back to the exact model layer and Python call path that triggered it. The question changes from "which GPU is expensive" to "which attention head is expensive".
- Per-layer GPU budget breakdown
- Identify synchronization idle time
- Memory bandwidth saturation per kernel
Built for production AI infrastructure
Multi-GPU support
Profile eight H100s or a multi-node inference cluster. Per-device flamegraphs and cross-node aggregation.
Framework agnostic
PyTorch, JAX, TensorFlow, and raw CUDA all appear in the same flamegraph. No per-framework plugins.
Historical comparison
Compare profiles from before and after a deploy. Find regressions that per-request timing averages hide.
Kernel-level attribution
See exactly which CUDA kernel consumed what percentage of total GPU time, traced back to the Python call that launched it.
Live dashboards
Real-time aggregated view of your inference fleet. Percentile breakdowns, anomaly highlights, and kernel hot paths.
Low overhead agent
Under 1% throughput overhead measured on H100 clusters running large-batch transformer inference.
Fits into your existing stack
zymtrace exports to Prometheus, sends alerts to your Slack or PagerDuty, and exposes a query API for custom dashboards.
Prometheus + Grafana
Export kernel-level metrics as Prometheus gauges. Build Grafana dashboards on top of the raw profiling data.
Alerts and notifications
Define threshold rules on kernel cost percentages. Route alerts to Slack, PagerDuty, or any webhook endpoint.
REST and Python API
Query profiling data programmatically. Build internal cost-allocation pipelines or CI-gated regression checks.
Kubernetes-native deployment
Deploy as a DaemonSet on your GPU nodes. Profiling starts automatically when pods launch, no per-pod configuration.
Connect your first GPU in 5 minutes
No credit card. No instrumentation. One command to start seeing your full GPU call path.