I wrote this deep dive to connect AMD MI300X hardware - XCDs, compute units, wavefronts, registers, and LDS to inference choices such as batching, KV-cache management, and kernel tiling.
Writing it helped me interpret GPU profiles more clearly: how work reaches execution hardware, where data is reused, and what limits progress.
Why does batching help inference? Why can a larger kernel tile make an operation slower?
The answers depend on how the GPU moves data, stores working values, and executes instructions. This article builds a mental model of MI300X execution to explain the hardware constraints behind these optimization choices.
Where does a compute unit fit inside MI300X?
A compute unit (CU) executes instructions that calculate results and move data.
MI300X contains eight compute chiplets, called Accelerator Complex Dies (XCDs). Each XCD has 40 physical CUs, with 38 active and two disabled. This gives the GPU 304 active CUs.
The 38 active CUs in each XCD share a 4 MB L2 cache. When the requested data is already in this cache, it can be supplied without another access to high-bandwidth memory(HBM).
Where is the data?
MI300X has eight stacks of high-bandwidth memory (HBM), with a total capacity of 192 GB. HBM holds model weights, KV-cache data, and other values needed during execution.
HBM is outside the CUs. The memory system moves data between HBM and the hardware that performs the calculations.
Keep these three locations in mind as you read further:
HBM holds data for GPU operations.
An XCD contains CUs and their shared L2 cache.
A CU executes instructions and holds registers and LDS.
Memory capacity tells us how much data fits. Memory bandwidth tells us how quickly data can move.
A model can fit in memory and still run slowly.
How does work reach a CU?
Software launches a kernel: a function designed to run across many GPU threads in parallel. Each thread executes its own instance of that function.
The kernel launch specifies two things:
The total number of threads.
How those threads are organized into workgroups.
This arrangement is known as launch geometry.
For example, a kernel launch might create 1,024 threads organized as four workgroups of 256 threads each.
The GPU assigns whole workgroups, not individual threads, to Compute Units (CUs). A workgroup can be assigned only when a CU has enough available resources, including wavefront slots, registers, and any required Local Data Share (LDS).
From Workgroups to Wavefronts
Within each workgroup, threads are organized into wavefronts. On MI300X, a wavefront contains 64 threads. Each thread has its own vector-register values and can access different memory locations.
A 256-thread workgroup therefore contains four wavefronts:
Wavefront 0: Threads 0–63
Wavefront 1: Threads 64–127
Wavefront 2: Threads 128–191
Wavefront 3: Threads 192–255
All four wavefronts from this workgroup execute on the same CU. However, they do not necessarily execute instructions at the same time. The CU schedules instructions from its ready wavefronts as execution resources become available.
How the CU Performs Calculations
Each MI300X CU contains four SIMD units. SIMD stands for Single Instruction, Multiple Data: one vector instruction is applied to values belonging to multiple threads.
For example, a wavefront can execute the same addition instruction across many threads:
Thread 0: 2 + 5 = 7
Thread 1: 7 + 1 = 8
Thread 2: 1 + 4 = 5
The instruction is the same - add two values but each thread supplies different inputs and receives its own result. The wavefront’s per-thread working values are stored in vector registers.
The 64-thread wavefront describes the group of threads that shares an instruction stream. It does not mean that all 64 thread results are physically produced in a single clock cycle.
The CU also includes hardware for other instruction types:
Scalar hardware handles calculations shared by the whole wavefront, such as updating a loop counter.
Matrix hardware multiplies blocks of numbers and adds the results to running totals, with threads working together.
Memory hardware loads and stores data. It accesses the memory hierarchy and also provides access to LDS, the CU’s shared local memory.
The CU runs an instruction when its data is ready and the hardware it needs is available. If one wavefront is waiting, the CU can run an instruction from another ready wavefront instead.
The overall flow is:
Software defines the work → the GPU places workgroups on CUs → each CU runs instructions from those workgroups’ wavefronts.
Why does this matter for inference?
GPU performance depends on both compute capacity and data movement. More execution hardware does not help if the GPU cannot receive data fast enough or does not have enough parallel work.
During LLM decode with a small batch, the GPU can spend significant time reading model weights from HBM. Batching improves weight reuse because the same weights can be used across more token rows.
More active requests also require more KV-cache memory. Efficient KV-cache management can leave room for more concurrent requests and create more opportunities for batching.
Larger tiles let the GPU reuse more data, but they also take up more space in registers and LDS. Tile size is a trade-off between data reuse and the CU’s limited storage.
Other optimizations reduce data movement. Kernel fusion combines operations so intermediate results do not need to be written to HBM and read back. Lower precision stores numbers using fewer bytes, so less data needs to move.
The goal is the same: move less data, reuse data when possible, and keep enough parallel work available to the GPU.
References :
1. AMD — MI300 Series microarchitecture
https://rocmdocs.amd.com/en/docs-7.2.4/conceptual/gpu-arch/mi300.html
2.AMD — CDNA 3 architecture white paper
https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/amd-cdna-3-white-paper.pdf
3.AMD — MI300X GPU partitioning overview
https://instinct.docs.amd.com/projects/amdgpu-docs/en/latest/gpu-partitioning/mi300x/overview.html
4.AMD HIP — Hardware implementation
https://rocm.docs.amd.com/projects/HIP/en/latest/understand/hardware_implementation.html
5.AMD HIP — Hardware features
https://rocm.docs.amd.com/projects/HIP/en/docs-7.0.2/reference/hardware_features.html
6.AMD ROCm Compute Profiler — Compute unit
https://rocm.docs.amd.com/projects/rocprofiler-compute/en/docs-10.0.0/conceptual/cdna/compute-unit.html
7.AMD ROCm Compute Profiler — Pipeline descriptions
https://rocm.docs.amd.com/projects/rocprofiler-compute/en/docs-7.14.1/conceptual/cdna/pipeline-descriptions.html
8.AMD — Matrix Core Programming on CDNA 3 and CDNA 4
https://rocm.blogs.amd.com/software-tools-optimization/matrix-cores-cdna/README.html
9.AMD HIP — Understanding GPU performance
https://rocm.docs.amd.com/projects/HIP/en/latest/understand/performance_optimization.html
10.AMD HIP — Performance guidelines
https://rocm.docs.amd.com/projects/HIP/en/latest/how-to/performance_guidelines.html
11.AMD — MI300/MI350 workload optimization
https://rocm.docs.amd.com/en/docs-7.2.4/how-to/rocm-for-ai/inference-optimization/workload.html
12.vLLM — PagedAttention introduction
https://vllm.ai/blog/2023-06-20-vllm
13.PagedAttention research paper — Efficient Memory Management for Large Language Models Serving with PagedAttention
https://doi.org/10.1145/3600006.3613165