# GPU Performance Optimization ## GPU Basics ### Thread Hierarchy ``` Grid → Blocks → Threads ``` - **Grid**: Entire launch, can be 1D/2D/3D - **Block**: Group of threads, max 1024 threads per block - **Thread**: Smallest unit, executes kernel function - **Warp**: 32 threads, executes same instruction simultaneously ### CPU vs GPU | Component | CPU | GPU | | ---------------- | ----------------- | ------------------------ | | Cores | Few (2-32) | Thousands | | Memory | RAM (large, slow) | VRAM (large, slow) | | On-chip | L1/L2/L3 cache | Shared memory, registers | | Thread switching | Slow | Fast (hardware) | | Best for | Serial, branching | Parallel, SIMT | ### When GPU is Faster - Large matrices (>~500 elements) - Data-parallel problems - Compute-bound operations - Not: small problems, branching-heavy code --- ## Memory Hierarchy ### Global Memory (VRAM) - Off-chip, high latency (~400 cycles) - Bandwidth-limited - Use for: input/output matrices ### Shared Memory - On-chip, low latency (~4 cycles) - Per-block, threads in block can share - Use for: tiles, reducing global accesses ### Registers - Per-thread, fastest - Trade parallelism for more registers ### Memory Access Pattern | Type | Speed | Use Case | | ----------------- | ------ | --------------- | | Global → Register | Slow | Input/output | | Global → Shared | Medium | Tiled algorithm | | Shared → Register | Fast | Computation | --- ## Coalesced Memory Access **Critical**: Threads in a warp should access consecutive memory addresses. ### Warp-Aligned Access ```cuda // GOOD: Warp accesses consecutive elements int row = blockIdx.y * blockDim.y + threadIdx.y; int col = blockIdx.x * blockDim.x + threadIdx.x; C[row * N + col] = ...; // Consecutive for threadIdx.x ``` ### Non-Coalesced (Slow) ```cuda // BAD: Strided access within warp // threadIdx.x accesses element at column = threadIdx.x * stride ``` ### Rule - Access memory in the same direction as storage layout (row-major) - Threads in a warp should have consecutive indices --- ## Tiled Matrix Multiplication ### Naive Approach (Slow) ``` Each thread: load row of A + column of B → compute dot product Problem: Each element loaded multiple times from global memory ``` ### Tiled Approach (Fast) 1. Divide matrices into tiles (match block size) 2. Load tile into shared memory 3. All threads in block use shared memory 4. Process next tile, accumulate results ### Code Pattern ```cuda __global__ void tiled_matmul(float* A, float* B, float* C, int N) { __shared__ float As[32][32]; // Tile in shared memory __shared__ float Bs[32][32]; int row = blockIdx.y * 32 + threadIdx.y; int col = blockIdx.x * 32 + threadIdx.x; float sum = 0; for (int k = 0; k < N; k += 32) { // Load tile into shared memory As[threadIdx.y][threadIdx.x] = A[row * N + (k + threadIdx.x)]; Bs[threadIdx.y][threadIdx.x] = B[(k + threadIdx.y) * N + col]; __syncthreads(); // Compute using shared memory (fast) for (int i = 0; i < 32; i++) sum += As[threadIdx.y][i] * Bs[i][threadIdx.x]; __syncthreads(); } C[row * N + col] = sum; } ``` ### Benefits - Reduces global memory accesses significantly - Threads in block share data via shared memory - Enable more compute per memory access --- ## Registers vs Parallelism ### Trade-off - More threads = more parallelism - Each thread needs registers - Registers per SM are limited ### Strategy Instead of 1 thread → 1 output element: - 1 thread → multiple output elements - Use registers to hold intermediate results - Reduce shared memory accesses ### Example ```cuda // Before: 1 thread → 1 element // After: 1 thread → 4 elements (2x2 tile) ``` --- ## Vectorization ### Concept Access multiple elements with single instruction: ```cuda // Before: 4 separate loads float a0 = A[idx + 0]; float a1 = A[idx + 1]; float a2 = A[idx + 2]; float a3 = A[idx + 3]; // After: single vector load (float4) float4 a = *(float4*)&A[idx]; ``` ### Benefits - Fewer instructions - Better memory bandwidth utilization --- ## Control Divergence ### Problem When threads in a warp take different paths: ```cuda if (condition) { // Different threads may take different paths // Path A } else { // Path B } ``` GPU executes both paths, disabling inactive threads. ### Mitigation - Restructure code to minimize branches - Use warp-level primitives (`__shfl`) - Avoid in hot paths --- ## Latency Hiding ### Concept GPU switches between warps while waiting for memory: - Memory latency: ~400 cycles - Warp switch: ~0 cycles (hardware) ### Requirements - Many warps per SM (occupancy) - Independent work ### Formula ``` Effective latency = latency / num_warps ``` --- ## Performance Analysis ### Roofline Model 1. Calculate operational intensity: FLOPs/byte 2. Find瓶颈: memory bandwidth or compute ceiling 3. Optimize accordingly ### Memory-Bound vs Compute-Bound ```c // Memory-bound: few FLOPs per memory access // Example: vector add c[i] = a[i] + b[i]; // 2 loads, 1 store, 1 FLOP // Compute-bound: many FLOPs per memory access // Example: matrix multiply (tiled) c[i][j] += a[i][k] * b[k][j]; // More compute per load ``` ### Bandwidth Calculation ``` Bandwidth = bytes_transferred / time Effective_FLOPS = bandwidth * operational_intensity ``` --- ## Checklist - [ ] Use coalesced memory access (consecutive threads → consecutive addresses) - [ ] Use shared memory for tiles - [ ] Match block size to tile size - [ ] Avoid control divergence in hot paths - [ ] Balance registers vs parallelism - [ ] Use vector loads/stores where possible - [ ] Profile with nvprof/nsys - [ ] Check memory bandwidth utilization - [ ] Ensure sufficient occupancy (warps/SM) - [ ] Hide latency with independent work --- ## Common Pitfalls 1. **Non-coalesced memory access**: Huge performance loss 2. **Too small blocks**: Low occupancy, no latency hiding 3. **No shared memory usage**: Global memory is bottleneck 4. **Control divergence**: Warps execute both paths 5. **Memory-bound**: Not enough compute per memory access 6. **Small problem sizes**: CPU may be faster --- ## Optimization Progression | Stage | Speedup | Technique | | ----- | -------- | ------------------- | | 1 | 1x | Naive parallel | | 2 | ~7x | Coalesced memory | | 3 | ~2x more | Shared memory tiles | | 4 | ~4x more | Register blocking | | 5 | ~2x more | Vectorization | | Total | ~70x | vs naive | --- ## Profiling Tools - **nvprof**: Legacy profiler - **nsys** (NVIDIA Nsight Systems): Timeline analysis - **nvvp**: Visual profiler - **CUDA Events**: Custom timing ```bash nsys profile ./my_program ```