GPU profiling, CUDA kernel analysis, inference cost engineering
Long-form technical writing from the zymtrace team. We publish what we learn building continuous GPU profiling infrastructure for AI inference at scale.
The CUDA Kernel Launch Latency Problem: Why Your Attention Layer Is Slower Than the Math Says
Kernel launch overhead compounds invisibly. We trace the path from Python dispatch to first GPU thread and show where the microseconds go.
The Hidden Cost of Kernel Fusion in PyTorch: When torch.compile Makes Things Slower
Kernel fusion is supposed to speed things up. Under certain batch shapes, it does the opposite. How to detect fusion regressions with flamegraphs.
Continuous Profiling vs Sampling: Why the Distinction Matters for AI Inference Workloads
The GPU utilization number your ops team is watching tells you almost nothing about where time is being wasted. The conceptual shift that changes inference optimization.
How We Cut GPU Cloud Spend by 31% Using Kernel-Level Attribution
Broad utilization optimization has diminishing returns. The companies making the biggest cuts do it at the kernel level, eliminating specific expensive call paths.
Designing a Continuous Profiling System for a Multi-Node GPU Inference Cluster
How zymtrace correlates call paths across 32 A100 nodes without introducing cross-node synchronization overhead.
Memory Copy Stalls on A100: The 8% of GPU Time Nobody Talks About
Host-to-device copy operations serialize with kernel execution in ways standard metrics do not surface. Continuous flamegraphs show this pattern immediately.
Reading a Flamegraph Without Falling Into the Histogram Trap
Flamegraph width represents cumulative time, not individual call duration. This distinction prevents the most common misread that leads teams to optimize the wrong function.
Tracing Python to CUDA: A Practical Guide to Full-Stack GPU Call Attribution
How a Python function call becomes a CUDA kernel dispatch, and where you can observe the transition. The conceptual model before any GPU profiling tool becomes useful.
GPU Profiling vs CPU Profiling for AI Inference: Why You Need Both and How They Relate
CPU profiles show Python call stacks. GPU profiles show kernel timelines. Neither alone tells you what is expensive. The intersection is where useful information lives.
Anatomy of a CUDA Kernel Launch in Flamegraphs
What each layer of a GPU flamegraph frame represents, from PyTorch operator dispatch down to the SM warp scheduler. A reference for interpreting zymtrace output for the first time.