repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by bugabinga

opencode/skills/performance-optimization/performance-gpu.md

Raw

GPU Performance Optimization

GPU Basics

Thread Hierarchy

Grid → Blocks → Threads

CPU vs GPU

Component CPU GPU
Cores Few (2-32) Thousands
Memory RAM (large, slow) VRAM (large, slow)
On-chip L1/L2/L3 cache Shared memory, registers
Thread switching Slow Fast (hardware)
Best for Serial, branching Parallel, SIMT

When GPU is Faster


Memory Hierarchy

Global Memory (VRAM)

Shared Memory

Registers

Memory Access Pattern

Type Speed Use Case
Global → Register Slow Input/output
Global → Shared Medium Tiled algorithm
Shared → Register Fast Computation

Coalesced Memory Access

Critical: Threads in a warp should access consecutive memory addresses.

Warp-Aligned Access

// GOOD: Warp accesses consecutive elements
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
C[row * N + col] = ...;  // Consecutive for threadIdx.x

Non-Coalesced (Slow)

// BAD: Strided access within warp
// threadIdx.x accesses element at column = threadIdx.x * stride

Rule


Tiled Matrix Multiplication

Naive Approach (Slow)

Each thread: load row of A + column of B → compute dot product
Problem: Each element loaded multiple times from global memory

Tiled Approach (Fast)

  1. Divide matrices into tiles (match block size)
  2. Load tile into shared memory
  3. All threads in block use shared memory
  4. Process next tile, accumulate results

Code Pattern

__global__ void tiled_matmul(float* A, float* B, float* C, int N) {
    __shared__ float As[32][32];  // Tile in shared memory
    __shared__ float Bs[32][32];
    
    int row = blockIdx.y * 32 + threadIdx.y;
    int col = blockIdx.x * 32 + threadIdx.x;
    
    float sum = 0;
    for (int k = 0; k < N; k += 32) {
        // Load tile into shared memory
        As[threadIdx.y][threadIdx.x] = A[row * N + (k + threadIdx.x)];
        Bs[threadIdx.y][threadIdx.x] = B[(k + threadIdx.y) * N + col];
        __syncthreads();
        
        // Compute using shared memory (fast)
        for (int i = 0; i < 32; i++)
            sum += As[threadIdx.y][i] * Bs[i][threadIdx.x];
        __syncthreads();
    }
    C[row * N + col] = sum;
}

Benefits


Registers vs Parallelism

Trade-off

Strategy

Instead of 1 thread → 1 output element:

Example

// Before: 1 thread → 1 element
// After: 1 thread → 4 elements (2x2 tile)

Vectorization

Concept

Access multiple elements with single instruction:

// Before: 4 separate loads
float a0 = A[idx + 0];
float a1 = A[idx + 1];
float a2 = A[idx + 2];
float a3 = A[idx + 3];

// After: single vector load (float4)
float4 a = *(float4*)&A[idx];

Benefits


Control Divergence

Problem

When threads in a warp take different paths:

if (condition) {  // Different threads may take different paths
    // Path A
} else {
    // Path B
}

GPU executes both paths, disabling inactive threads.

Mitigation


Latency Hiding

Concept

GPU switches between warps while waiting for memory:

Requirements

Formula

Effective latency = latency / num_warps

Performance Analysis

Roofline Model

  1. Calculate operational intensity: FLOPs/byte
  2. Find瓶颈: memory bandwidth or compute ceiling
  3. Optimize accordingly

Memory-Bound vs Compute-Bound

// Memory-bound: few FLOPs per memory access
// Example: vector add
c[i] = a[i] + b[i];  // 2 loads, 1 store, 1 FLOP

// Compute-bound: many FLOPs per memory access  
// Example: matrix multiply (tiled)
c[i][j] += a[i][k] * b[k][j];  // More compute per load

Bandwidth Calculation

Bandwidth = bytes_transferred / time
Effective_FLOPS = bandwidth * operational_intensity

Checklist


Common Pitfalls

  1. Non-coalesced memory access: Huge performance loss
  2. Too small blocks: Low occupancy, no latency hiding
  3. No shared memory usage: Global memory is bottleneck
  4. Control divergence: Warps execute both paths
  5. Memory-bound: Not enough compute per memory access
  6. Small problem sizes: CPU may be faster

Optimization Progression

Stage Speedup Technique
1 1x Naive parallel
2 ~7x Coalesced memory
3 ~2x more Shared memory tiles
4 ~4x more Register blocking
5 ~2x more Vectorization
Total ~70x vs naive

Profiling Tools

nsys profile ./my_program
Markdown source
# GPU Performance Optimization

## GPU Basics

### Thread Hierarchy

```
Grid → Blocks → Threads
```

- **Grid**: Entire launch, can be 1D/2D/3D
- **Block**: Group of threads, max 1024 threads per block
- **Thread**: Smallest unit, executes kernel function
- **Warp**: 32 threads, executes same instruction simultaneously

### CPU vs GPU

| Component        | CPU               | GPU                      |
| ---------------- | ----------------- | ------------------------ |
| Cores            | Few (2-32)        | Thousands                |
| Memory           | RAM (large, slow) | VRAM (large, slow)       |
| On-chip          | L1/L2/L3 cache    | Shared memory, registers |
| Thread switching | Slow              | Fast (hardware)          |
| Best for         | Serial, branching | Parallel, SIMT           |

### When GPU is Faster

- Large matrices (>~500 elements)
- Data-parallel problems
- Compute-bound operations
- Not: small problems, branching-heavy code

---

## Memory Hierarchy

### Global Memory (VRAM)

- Off-chip, high latency (~400 cycles)
- Bandwidth-limited
- Use for: input/output matrices

### Shared Memory

- On-chip, low latency (~4 cycles)
- Per-block, threads in block can share
- Use for: tiles, reducing global accesses

### Registers

- Per-thread, fastest
- Trade parallelism for more registers

### Memory Access Pattern

| Type              | Speed  | Use Case        |
| ----------------- | ------ | --------------- |
| Global → Register | Slow   | Input/output    |
| Global → Shared   | Medium | Tiled algorithm |
| Shared → Register | Fast   | Computation     |

---

## Coalesced Memory Access

**Critical**: Threads in a warp should access consecutive memory addresses.

### Warp-Aligned Access

```cuda
// GOOD: Warp accesses consecutive elements
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
C[row * N + col] = ...;  // Consecutive for threadIdx.x
```

### Non-Coalesced (Slow)

```cuda
// BAD: Strided access within warp
// threadIdx.x accesses element at column = threadIdx.x * stride
```

### Rule

- Access memory in the same direction as storage layout (row-major)
- Threads in a warp should have consecutive indices

---

## Tiled Matrix Multiplication

### Naive Approach (Slow)

```
Each thread: load row of A + column of B → compute dot product
Problem: Each element loaded multiple times from global memory
```

### Tiled Approach (Fast)

1. Divide matrices into tiles (match block size)
2. Load tile into shared memory
3. All threads in block use shared memory
4. Process next tile, accumulate results

### Code Pattern

```cuda
__global__ void tiled_matmul(float* A, float* B, float* C, int N) {
    __shared__ float As[32][32];  // Tile in shared memory
    __shared__ float Bs[32][32];
    
    int row = blockIdx.y * 32 + threadIdx.y;
    int col = blockIdx.x * 32 + threadIdx.x;
    
    float sum = 0;
    for (int k = 0; k < N; k += 32) {
        // Load tile into shared memory
        As[threadIdx.y][threadIdx.x] = A[row * N + (k + threadIdx.x)];
        Bs[threadIdx.y][threadIdx.x] = B[(k + threadIdx.y) * N + col];
        __syncthreads();
        
        // Compute using shared memory (fast)
        for (int i = 0; i < 32; i++)
            sum += As[threadIdx.y][i] * Bs[i][threadIdx.x];
        __syncthreads();
    }
    C[row * N + col] = sum;
}
```

### Benefits

- Reduces global memory accesses significantly
- Threads in block share data via shared memory
- Enable more compute per memory access

---

## Registers vs Parallelism

### Trade-off

- More threads = more parallelism
- Each thread needs registers
- Registers per SM are limited

### Strategy

Instead of 1 thread → 1 output element:

- 1 thread → multiple output elements
- Use registers to hold intermediate results
- Reduce shared memory accesses

### Example

```cuda
// Before: 1 thread → 1 element
// After: 1 thread → 4 elements (2x2 tile)
```

---

## Vectorization

### Concept

Access multiple elements with single instruction:

```cuda
// Before: 4 separate loads
float a0 = A[idx + 0];
float a1 = A[idx + 1];
float a2 = A[idx + 2];
float a3 = A[idx + 3];

// After: single vector load (float4)
float4 a = *(float4*)&A[idx];
```

### Benefits

- Fewer instructions
- Better memory bandwidth utilization

---

## Control Divergence

### Problem

When threads in a warp take different paths:

```cuda
if (condition) {  // Different threads may take different paths
    // Path A
} else {
    // Path B
}
```

GPU executes both paths, disabling inactive threads.

### Mitigation

- Restructure code to minimize branches
- Use warp-level primitives (`__shfl`)
- Avoid in hot paths

---

## Latency Hiding

### Concept

GPU switches between warps while waiting for memory:

- Memory latency: ~400 cycles
- Warp switch: ~0 cycles (hardware)

### Requirements

- Many warps per SM (occupancy)
- Independent work

### Formula

```
Effective latency = latency / num_warps
```

---

## Performance Analysis

### Roofline Model

1. Calculate operational intensity: FLOPs/byte
2. Find瓶颈: memory bandwidth or compute ceiling
3. Optimize accordingly

### Memory-Bound vs Compute-Bound

```c
// Memory-bound: few FLOPs per memory access
// Example: vector add
c[i] = a[i] + b[i];  // 2 loads, 1 store, 1 FLOP

// Compute-bound: many FLOPs per memory access  
// Example: matrix multiply (tiled)
c[i][j] += a[i][k] * b[k][j];  // More compute per load
```

### Bandwidth Calculation

```
Bandwidth = bytes_transferred / time
Effective_FLOPS = bandwidth * operational_intensity
```

---

## Checklist

- [ ] Use coalesced memory access (consecutive threads → consecutive addresses)
- [ ] Use shared memory for tiles
- [ ] Match block size to tile size
- [ ] Avoid control divergence in hot paths
- [ ] Balance registers vs parallelism
- [ ] Use vector loads/stores where possible
- [ ] Profile with nvprof/nsys
- [ ] Check memory bandwidth utilization
- [ ] Ensure sufficient occupancy (warps/SM)
- [ ] Hide latency with independent work

---

## Common Pitfalls

1. **Non-coalesced memory access**: Huge performance loss
2. **Too small blocks**: Low occupancy, no latency hiding
3. **No shared memory usage**: Global memory is bottleneck
4. **Control divergence**: Warps execute both paths
5. **Memory-bound**: Not enough compute per memory access
6. **Small problem sizes**: CPU may be faster

---

## Optimization Progression

| Stage | Speedup  | Technique           |
| ----- | -------- | ------------------- |
| 1     | 1x       | Naive parallel      |
| 2     | ~7x      | Coalesced memory    |
| 3     | ~2x more | Shared memory tiles |
| 4     | ~4x more | Register blocking   |
| 5     | ~2x more | Vectorization       |
| Total | ~70x     | vs naive            |

---

## Profiling Tools

- **nvprof**: Legacy profiler
- **nsys** (NVIDIA Nsight Systems): Timeline analysis
- **nvvp**: Visual profiler
- **CUDA Events**: Custom timing

```bash
nsys profile ./my_program
```