Reading GPU Kernel Traces at the Hardware Level

When an LLM request is slow, application metrics tell you how slow it is but not which GPU operations deserve a closer look.

AMD’s command-line profiler, rocprofv3, plays a similar role to NVIDIA’s Nsight profiling tools: it helps you look beneath application metrics to understand GPU execution. It shows which kernels ran, how often they launched, and how much GPU time they accumulated. The point is not to collect more numbers. It is to turn “the request is slow” into a specific question we can investigate.

In this post, I trace a Qwen3 MoE inference request on an AMD MI300X and explain why one kernel family ran 6,144 times. By connecting launch counts and timestamps to the model’s layers and decode steps, you’ll see how to turn thousands of dispatches into an understandable execution pattern.

That gives us a starting point for deeper profiling: separating the initial pass from repeated decode work and identifying the kernels for counter collection in Part 2. In Part 3, we convert those counters into measurements of arithmetic and data movement.

When an inference request runs, the GPU executes many small programs called kernels. Some perform matrix multiplication. Others compute attention, select experts, or update the KV cache.

Each execution of a kernel is a dispatch. Here, “kernel” means a GPU program, not the Linux operating-system kernel.

Application metrics show request latency and token generation speed. A kernel trace lets us look inside that execution: what ran, how often it ran, and how long each dispatch took.

I used AMD’s rocprofv3 tool to investigate Qwen3-30B-A3B inference on an MI300X. The measured request produced 54,194 GPU dispatches. The first task was to turn that list into an understandable execution pattern.

The experiment used the following configuration:

There was no HTTP server in this path. The Python program called vLLM directly.

The model was loaded and warmed up before profiling started. The script synchronized GPU work before resuming profiling and again before pausing it. The profiler resume/pause calls defined the measured region, keeping model loading and warm-up outside it.

Synchronization does not prove that every possible lazy initialization had already happened. But it makes the measured region clearer.

The 64 output tokens were generated one at a time. They were not processed together as a batch of 64 during decode. This matters when interpreting the hardware measurements later.

The first pass collected a kernel trace and aggregate statistics. A simplified command from the experiment was:

The script measures the generation of exactly 64 output tokens. The trace CSV records each GPU dispatch during that generation. The statistics CSV summarizes how often each kernel ran and how much time it used.

The summary gave a starting point:

The 25.72% is a share of summed kernel duration. It is not a claim that this family accounts for exactly 25.72% of request latency. CPU work, synchronization, gaps, and overlapping execution affect the full request.

Even with that distinction, the family was a significant enough contributor to investigate.

A shared kernel name does not mean every launch has the same shape.

Four IDs appeared within this family:

These IDs are labels from this trace, not permanent identifiers to search for in every run.

All four entries reported 80 VGPRs, 48 SGPRs, zero LDS allocation, and zero scratch allocation. Those resource fields help describe the launches; they do not establish occupancy or a bottleneck by themselves. Zero LDS allocation also does not mean the kernel moves no data.

For these wave64 launches, a 256-thread workgroup contains:

A wavefront is a group of threads that execute instructions together. 

This describes how work is organized within a workgroup. The dispatch counts answer a different question: how often did the model need this work?

Qwen3-30B-A3B has 48 layers. The initial pass produces the first output token, followed by 63 single-token decode steps.

The two variants associated with the initial pass appeared 48 times each. The two repeated decode variants appeared 3,024 times each:

63 decode steps × 48 layers = 3,024 launches per variant

Together:

48 + 48 + 3,024 + 3,024 = 6,144 launches

These counts are consistent with two MoE kernel launches per layer during the initial pass and each subsequent decode step.

The trace showed what ran and how often. But it did not show what the GPU was doing inside those kernels.

How much matrix work did these repeated decode kernels perform, and how much data did they move?

In Part 2, we select and group the hardware counters needed to investigate that question, working around a practical constraint: the GPU cannot collect all the requested counters at the same time.

Code: Explore the workload script, raw captures, runbook, and reproducible analysis in the MI300X MoE profiling project.