dotfiles
bugabingas dorkfiles
owned by bugabinga
opencode/skills/performance-optimization/performance-gpu.md
RawGPU Performance Optimization
GPU Basics
Thread Hierarchy
Grid → Blocks → Threads
- Grid: Entire launch, can be 1D/2D/3D
- Block: Group of threads, max 1024 threads per block
- Thread: Smallest unit, executes kernel function
- Warp: 32 threads, executes same instruction simultaneously
CPU vs GPU
| Component |
CPU |
GPU |
| Cores |
Few (2-32) |
Thousands |
| Memory |
RAM (large, slow) |
VRAM (large, slow) |
| On-chip |
L1/L2/L3 cache |
Shared memory, registers |
| Thread switching |
Slow |
Fast (hardware) |
| Best for |
Serial, branching |
Parallel, SIMT |
When GPU is Faster
- Large matrices (>~500 elements)
- Data-parallel problems
- Compute-bound operations
- Not: small problems, branching-heavy code
Memory Hierarchy
Global Memory (VRAM)
- Off-chip, high latency (~400 cycles)
- Bandwidth-limited
- Use for: input/output matrices
Shared Memory
- On-chip, low latency (~4 cycles)
- Per-block, threads in block can share
- Use for: tiles, reducing global accesses
Registers
- Per-thread, fastest
- Trade parallelism for more registers
Memory Access Pattern
| Type |
Speed |
Use Case |
| Global → Register |
Slow |
Input/output |
| Global → Shared |
Medium |
Tiled algorithm |
| Shared → Register |
Fast |
Computation |
Coalesced Memory Access
Critical: Threads in a warp should access consecutive memory addresses.
Warp-Aligned Access
// GOOD: Warp accesses consecutive elements
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
C[row * N + col] = ...; // Consecutive for threadIdx.x
Non-Coalesced (Slow)
// BAD: Strided access within warp
// threadIdx.x accesses element at column = threadIdx.x * stride
Rule
- Access memory in the same direction as storage layout (row-major)
- Threads in a warp should have consecutive indices
Tiled Matrix Multiplication
Naive Approach (Slow)
Each thread: load row of A + column of B → compute dot product
Problem: Each element loaded multiple times from global memory
Tiled Approach (Fast)
- Divide matrices into tiles (match block size)
- Load tile into shared memory
- All threads in block use shared memory
- Process next tile, accumulate results
Code Pattern
__global__ void tiled_matmul(float* A, float* B, float* C, int N) {
__shared__ float As[32][32]; // Tile in shared memory
__shared__ float Bs[32][32];
int row = blockIdx.y * 32 + threadIdx.y;
int col = blockIdx.x * 32 + threadIdx.x;
float sum = 0;
for (int k = 0; k < N; k += 32) {
// Load tile into shared memory
As[threadIdx.y][threadIdx.x] = A[row * N + (k + threadIdx.x)];
Bs[threadIdx.y][threadIdx.x] = B[(k + threadIdx.y) * N + col];
__syncthreads();
// Compute using shared memory (fast)
for (int i = 0; i < 32; i++)
sum += As[threadIdx.y][i] * Bs[i][threadIdx.x];
__syncthreads();
}
C[row * N + col] = sum;
}
Benefits
- Reduces global memory accesses significantly
- Threads in block share data via shared memory
- Enable more compute per memory access
Registers vs Parallelism
Trade-off
- More threads = more parallelism
- Each thread needs registers
- Registers per SM are limited
Strategy
Instead of 1 thread → 1 output element:
- 1 thread → multiple output elements
- Use registers to hold intermediate results
- Reduce shared memory accesses
Example
// Before: 1 thread → 1 element
// After: 1 thread → 4 elements (2x2 tile)
Vectorization
Concept
Access multiple elements with single instruction:
// Before: 4 separate loads
float a0 = A[idx + 0];
float a1 = A[idx + 1];
float a2 = A[idx + 2];
float a3 = A[idx + 3];
// After: single vector load (float4)
float4 a = *(float4*)&A[idx];
Benefits
- Fewer instructions
- Better memory bandwidth utilization
Control Divergence
Problem
When threads in a warp take different paths:
if (condition) { // Different threads may take different paths
// Path A
} else {
// Path B
}
GPU executes both paths, disabling inactive threads.
Mitigation
- Restructure code to minimize branches
- Use warp-level primitives (
__shfl)
- Avoid in hot paths
Latency Hiding
Concept
GPU switches between warps while waiting for memory:
- Memory latency: ~400 cycles
- Warp switch: ~0 cycles (hardware)
Requirements
- Many warps per SM (occupancy)
- Independent work
Formula
Effective latency = latency / num_warps
Performance Analysis
Roofline Model
- Calculate operational intensity: FLOPs/byte
- Find瓶颈: memory bandwidth or compute ceiling
- Optimize accordingly
Memory-Bound vs Compute-Bound
// Memory-bound: few FLOPs per memory access
// Example: vector add
c[i] = a[i] + b[i]; // 2 loads, 1 store, 1 FLOP
// Compute-bound: many FLOPs per memory access
// Example: matrix multiply (tiled)
c[i][j] += a[i][k] * b[k][j]; // More compute per load
Bandwidth Calculation
Bandwidth = bytes_transferred / time
Effective_FLOPS = bandwidth * operational_intensity
Checklist
- Use coalesced memory access (consecutive threads → consecutive addresses)
- Use shared memory for tiles
- Match block size to tile size
- Avoid control divergence in hot paths
- Balance registers vs parallelism
- Use vector loads/stores where possible
- Profile with nvprof/nsys
- Check memory bandwidth utilization
- Ensure sufficient occupancy (warps/SM)
- Hide latency with independent work
Common Pitfalls
- Non-coalesced memory access: Huge performance loss
- Too small blocks: Low occupancy, no latency hiding
- No shared memory usage: Global memory is bottleneck
- Control divergence: Warps execute both paths
- Memory-bound: Not enough compute per memory access
- Small problem sizes: CPU may be faster
Optimization Progression
| Stage |
Speedup |
Technique |
| 1 |
1x |
Naive parallel |
| 2 |
~7x |
Coalesced memory |
| 3 |
~2x more |
Shared memory tiles |
| 4 |
~4x more |
Register blocking |
| 5 |
~2x more |
Vectorization |
| Total |
~70x |
vs naive |
Profiling Tools
- nvprof: Legacy profiler
- nsys (NVIDIA Nsight Systems): Timeline analysis
- nvvp: Visual profiler
- CUDA Events: Custom timing
nsys profile ./my_program
Markdown source
# GPU Performance Optimization
## GPU Basics
### Thread Hierarchy
```
Grid → Blocks → Threads
```
- **Grid**: Entire launch, can be 1D/2D/3D
- **Block**: Group of threads, max 1024 threads per block
- **Thread**: Smallest unit, executes kernel function
- **Warp**: 32 threads, executes same instruction simultaneously
### CPU vs GPU
| Component | CPU | GPU |
| ---------------- | ----------------- | ------------------------ |
| Cores | Few (2-32) | Thousands |
| Memory | RAM (large, slow) | VRAM (large, slow) |
| On-chip | L1/L2/L3 cache | Shared memory, registers |
| Thread switching | Slow | Fast (hardware) |
| Best for | Serial, branching | Parallel, SIMT |
### When GPU is Faster
- Large matrices (>~500 elements)
- Data-parallel problems
- Compute-bound operations
- Not: small problems, branching-heavy code
---
## Memory Hierarchy
### Global Memory (VRAM)
- Off-chip, high latency (~400 cycles)
- Bandwidth-limited
- Use for: input/output matrices
### Shared Memory
- On-chip, low latency (~4 cycles)
- Per-block, threads in block can share
- Use for: tiles, reducing global accesses
### Registers
- Per-thread, fastest
- Trade parallelism for more registers
### Memory Access Pattern
| Type | Speed | Use Case |
| ----------------- | ------ | --------------- |
| Global → Register | Slow | Input/output |
| Global → Shared | Medium | Tiled algorithm |
| Shared → Register | Fast | Computation |
---
## Coalesced Memory Access
**Critical**: Threads in a warp should access consecutive memory addresses.
### Warp-Aligned Access
```cuda
// GOOD: Warp accesses consecutive elements
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
C[row * N + col] = ...; // Consecutive for threadIdx.x
```
### Non-Coalesced (Slow)
```cuda
// BAD: Strided access within warp
// threadIdx.x accesses element at column = threadIdx.x * stride
```
### Rule
- Access memory in the same direction as storage layout (row-major)
- Threads in a warp should have consecutive indices
---
## Tiled Matrix Multiplication
### Naive Approach (Slow)
```
Each thread: load row of A + column of B → compute dot product
Problem: Each element loaded multiple times from global memory
```
### Tiled Approach (Fast)
1. Divide matrices into tiles (match block size)
2. Load tile into shared memory
3. All threads in block use shared memory
4. Process next tile, accumulate results
### Code Pattern
```cuda
__global__ void tiled_matmul(float* A, float* B, float* C, int N) {
__shared__ float As[32][32]; // Tile in shared memory
__shared__ float Bs[32][32];
int row = blockIdx.y * 32 + threadIdx.y;
int col = blockIdx.x * 32 + threadIdx.x;
float sum = 0;
for (int k = 0; k < N; k += 32) {
// Load tile into shared memory
As[threadIdx.y][threadIdx.x] = A[row * N + (k + threadIdx.x)];
Bs[threadIdx.y][threadIdx.x] = B[(k + threadIdx.y) * N + col];
__syncthreads();
// Compute using shared memory (fast)
for (int i = 0; i < 32; i++)
sum += As[threadIdx.y][i] * Bs[i][threadIdx.x];
__syncthreads();
}
C[row * N + col] = sum;
}
```
### Benefits
- Reduces global memory accesses significantly
- Threads in block share data via shared memory
- Enable more compute per memory access
---
## Registers vs Parallelism
### Trade-off
- More threads = more parallelism
- Each thread needs registers
- Registers per SM are limited
### Strategy
Instead of 1 thread → 1 output element:
- 1 thread → multiple output elements
- Use registers to hold intermediate results
- Reduce shared memory accesses
### Example
```cuda
// Before: 1 thread → 1 element
// After: 1 thread → 4 elements (2x2 tile)
```
---
## Vectorization
### Concept
Access multiple elements with single instruction:
```cuda
// Before: 4 separate loads
float a0 = A[idx + 0];
float a1 = A[idx + 1];
float a2 = A[idx + 2];
float a3 = A[idx + 3];
// After: single vector load (float4)
float4 a = *(float4*)&A[idx];
```
### Benefits
- Fewer instructions
- Better memory bandwidth utilization
---
## Control Divergence
### Problem
When threads in a warp take different paths:
```cuda
if (condition) { // Different threads may take different paths
// Path A
} else {
// Path B
}
```
GPU executes both paths, disabling inactive threads.
### Mitigation
- Restructure code to minimize branches
- Use warp-level primitives (`__shfl`)
- Avoid in hot paths
---
## Latency Hiding
### Concept
GPU switches between warps while waiting for memory:
- Memory latency: ~400 cycles
- Warp switch: ~0 cycles (hardware)
### Requirements
- Many warps per SM (occupancy)
- Independent work
### Formula
```
Effective latency = latency / num_warps
```
---
## Performance Analysis
### Roofline Model
1. Calculate operational intensity: FLOPs/byte
2. Find瓶颈: memory bandwidth or compute ceiling
3. Optimize accordingly
### Memory-Bound vs Compute-Bound
```c
c[i] = a[i] + b[i];
c[i][j] += a[i][k] * b[k][j];
```
### Bandwidth Calculation
```
Bandwidth = bytes_transferred / time
Effective_FLOPS = bandwidth * operational_intensity
```
---
## Checklist
- [ ] Use coalesced memory access (consecutive threads → consecutive addresses)
- [ ] Use shared memory for tiles
- [ ] Match block size to tile size
- [ ] Avoid control divergence in hot paths
- [ ] Balance registers vs parallelism
- [ ] Use vector loads/stores where possible
- [ ] Profile with nvprof/nsys
- [ ] Check memory bandwidth utilization
- [ ] Ensure sufficient occupancy (warps/SM)
- [ ] Hide latency with independent work
---
## Common Pitfalls
1. **Non-coalesced memory access**: Huge performance loss
2. **Too small blocks**: Low occupancy, no latency hiding
3. **No shared memory usage**: Global memory is bottleneck
4. **Control divergence**: Warps execute both paths
5. **Memory-bound**: Not enough compute per memory access
6. **Small problem sizes**: CPU may be faster
---
## Optimization Progression
| Stage | Speedup | Technique |
| ----- | -------- | ------------------- |
| 1 | 1x | Naive parallel |
| 2 | ~7x | Coalesced memory |
| 3 | ~2x more | Shared memory tiles |
| 4 | ~4x more | Register blocking |
| 5 | ~2x more | Vectorization |
| Total | ~70x | vs naive |
---
## Profiling Tools
- **nvprof**: Legacy profiler
- **nsys** (NVIDIA Nsight Systems): Timeline analysis
- **nvvp**: Visual profiler
- **CUDA Events**: Custom timing
```bash
nsys profile ./my_program
```