Quickstart

Prerequisites

• A zymtrace account (free tier works). Create one here.
• Linux node with NVIDIA GPU (CUDA 11.6+, kernel 5.8+)
• Python 3.9-3.12, PyTorch 2.0+
• Your zymtrace API token (from the dashboard under Settings)

Step 1: Install the agent

Install the agent binary using the one-line installer. It will detect your CUDA version and kernel architecture automatically.

curl -fsSL https://install.gozymtrace.com/agent | bash

Or install via pip if you prefer a Python-managed install:

pip install zymtrace-agent

Step 2: Authenticate

Set your API token as an environment variable. You can find this in your dashboard under Settings > API Tokens.

export ZMT_API_TOKEN=your-api-token-here

For production, add this to your systemd unit file or container environment rather than exporting inline.

Step 3: Start the agent

Launch the agent, pointing it at the PID of your running inference process. If your process is not running yet, you can also start the agent first and attach on process start.

# Attach to a running process by PID
zymtrace attach --pid $(pgrep -f "python inference_server.py")

# Or start and auto-attach when your process launches
zymtrace watch --match "inference_server.py"

You should see output like:

zymtrace agent v0.9.2
Attaching to PID 18342 (python)
eBPF uprobes: loaded (python3.11, libcuda.so)
CUDA correlation: active
Sampling rate: 99Hz
Endpoint: api.gozymtrace.com:443 (TLS)
First batch queued at T+1.2s
Streaming... (Ctrl+C to detach)

Step 4: Open the dashboard

Navigate to app.gozymtrace.com and select your node from the sidebar. You will see the live flamegraph populating as call-path batches arrive from the agent.

The initial view shows the last 60 seconds. Use the time range selector to zoom in on specific intervals or expand to the last 24 hours.

Step 5: Read your flamegraph

The flamegraph width at each level represents cumulative GPU time spent in that call path. Wider bars are more expensive. Orange bars are CUDA kernel dispatches; blue-white bars are Python frames.

Click any bar to drill into that call path and see:

• Total GPU time over the selected window
• Average kernel duration per dispatch
• Dispatch frequency (how many times per second this kernel launched)
• GPU utilization while this kernel was active

Running in Kubernetes

Add the zymtrace agent as a sidecar container in your inference pod spec:

containers:
  - name: inference
    image: your-inference-image:latest
    # ... existing container config ...

  - name: zymtrace-agent
    image: ghcr.io/zymtrace/agent:latest
    env:
      - name: ZMT_API_TOKEN
        valueFrom:
          secretKeyRef:
            name: zymtrace-secret
            key: api-token
      - name: ZMT_WATCH_PATTERN
        value: "inference_server.py"
    securityContext:
      capabilities:
        add: ["BPF", "PERFMON"]
    volumeMounts:
      - name: proc
        mountPath: /host/proc
        readOnly: true

volumes:
  - name: proc
    hostPath:
      path: /proc

Next steps