Memory Profiling: torch.profiler, Nsight Systems, OOM Debugging — Production GPU Memory Yönetimi

Ders Haritası#

1. GPU memory anatomy — neler olabiliyor
2. Memory breakdown: weights, activations, gradients, optimizer
3. torch.cuda.memory_*
   API'leri
4. torch.profiler memory snapshot
5. Nsight Systems timeline analizi
6. OOM root cause patterns
7. Activation checkpointing
8. Gradient accumulation
9. Memory fragmentation
10. Production memory budget planning

1. GPU Memory Anatomy#

Static (sabit) memory#

• Model weights (BF16): N param × 2 byte
• Master weights (FP32): N × 4 byte
• Optimizer states (Adam: m + v): N × 8 byte (FP32)
• CUDA context: ~1 GB
• PyTorch overhead: ~0.5 GB

Dynamic (training adımına göre değişen) memory#

• Activations: forward pass'te saklananlar (backward için)
• Gradients (BF16): N × 2 byte
• Temporary buffers: matmul intermediate, layer outputs
• KV cache (inference): per-token cache

70B Llama 3 örneği (training)#

Weights BF16:       140 GB
Master FP32:        280 GB
Optimizer (Adam):   560 GB
Gradients BF16:     140 GB
Activations:         ~50 GB (batch 8, seq 8192)

Inference scenario (70B)#

Weights BF16:       140 GB
KV cache (1 token): ~2 MB/token, 8K context = 16 GB
Activations:        ~5 GB

2.

torch.cuda.memory_*

API'leri#

import torch

# Snapshot
allocated = torch.cuda.memory_allocated()        # şu an ayrılmış (bytes)
reserved = torch.cuda.memory_reserved()           # PyTorch'un toplam tuttuğu (bytes)
max_allocated = torch.cuda.max_memory_allocated()  # peak (training boyunca)
max_reserved = torch.cuda.max_memory_reserved()

# Reset peak stats
torch.cuda.reset_peak_memory_stats()

# Stats string
print(torch.cuda.memory_summary())

allocated

vs

reserved

#

• allocated: tensor'ların kullandığı bellek
• reserved: PyTorch'un caching allocator'unun cuda'dan istediği (cache dahil)

Pratik kullanım#

def report_memory(label):
    a = torch.cuda.memory_allocated() / 1024**3
    r = torch.cuda.memory_reserved() / 1024**3
    p = torch.cuda.max_memory_allocated() / 1024**3
    print(f"{label}: {a:.2f}GB allocated, {r:.2f}GB reserved, {p:.2f}GB peak")

report_memory("Before model load")
model = load_model().cuda()
report_memory("After model load")

x = torch.randn(32, 1024, device="cuda")
report_memory("After input creation")

out = model(x)
report_memory("After forward")

out.sum().backward()
report_memory("After backward")

3. torch.profiler — Memory Snapshot#

Basic usage#

from torch.profiler import profile, ProfilerActivity, record_function

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    profile_memory=True,
    record_shapes=True,
) as prof:
    out = model(x)
    out.sum().backward()

print(prof.key_averages().table(sort_by="self_cuda_memory_usage", row_limit=20))

Memory snapshot — tensor-level detay#

torch.cuda.memory._record_memory_history(enabled="all")

# Training step
out = model(x)
out.sum().backward()
optimizer.step()

torch.cuda.memory._dump_snapshot("memory_snapshot.pickle")
torch.cuda.memory._record_memory_history(enabled=None)

https://pytorch.org/memory_viz

Visualization#

• Allocation stack trace — hangi koddan
• Tensor lifetime — başlangıç ve bitiş
• Memory pressure peaks
• Reuse patterns

4. Nsight Systems — Timeline Analizi#

Kurulum#

nsys profile -o profile_out python train.py

Görselleştirilenler#

1. CUDA kernel timeline: hangi kernel ne zaman, ne kadar süre
2. CPU activity: Python overhead, data loading
3. Memcpy ops: CPU↔GPU transfers
4. NCCL collective ops: distributed training'de
5. GPU utilization: % busy
6. Power consumption

Tipik findings#

Pattern 1: Idle GPU

Pattern 2: Long kernels

Pattern 3: Frequent small kernels

Pattern 4: Communication overhead

Modern alternatif: NVIDIA DLProf#

5. OOM Root Cause Patterns#

Pattern 1: Batch size too large#

Pattern 2: Sequence too long#

Pattern 3: Activation explosion#

Pattern 4: Gradient retention#

Pattern 5: Multiple model copies#

Pattern 6: Memory fragmentation#

Pattern 7: Other tensors#

6. Activation Checkpointing — Memory ↔ Compute Trade-off#

Trade-off#

• Memory: O(N) → O(√N) (Chen 2016, gradient checkpointing)
• Compute: forward 1x + backward'da extra forward = ~1.5x compute total

PyTorch usage#

from torch.utils.checkpoint import checkpoint

class TransformerBlock(nn.Module):
    def forward(self, x):
        return checkpoint(self._forward, x, use_reentrant=False)

    def _forward(self, x):
        x = self.attention(x)
        x = self.ffn(x)
        return x

Hangi katmanlara?#

• Transformer blokları (en yaygın)
• Encoder/decoder ayrı ayrı
• Memory-heavy operations (attention, large FFN)

Selective checkpointing#

# Sadece her N. blok checkpoint
class Transformer(nn.Module):
    def forward(self, x):
        for i, block in enumerate(self.blocks):
            if i % 2 == 0:  # her ikinci blok checkpoint
                x = checkpoint(block, x, use_reentrant=False)
            else:
                x = block(x)
        return x

use_reentrant=False

niye?#

FSDP ile combine#

from torch.distributed.fsdp import FullyShardedDataParallel, checkpoint_wrapper

# Each FSDP module wrapped with checkpoint

7. Gradient Accumulation — Effective Batch Size Artırma#

Algoritma#

accumulation_steps = 4
optimizer.zero_grad()

for i, batch in enumerate(dataloader):
    with autocast(device_type="cuda", dtype=torch.bfloat16):
        out = model(batch.input)
        loss = criterion(out, batch.target) / accumulation_steps  # scale
    loss.backward()

    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Mathematics#

• micro_batch=4, accumulation=4 → effective_batch=16
• Aynı loss landscape (batch=16 ile equivalent)
• Memory: micro_batch boyutu (4) kadar, gradient sum cumulative ama tek gradient buffer

Trade-off#

• ✓ Memory tasarruf
• ✗ Speed yavaş (her step birden çok forward+backward)
• ✓ Gradient quality eşit

Pratik#

• Naive batch=8: OOM
• Micro batch=2 + accumulation=4: OK (effective batch=8)

FSDP/ZeRO ile combine#

# FSDP shards parameters across GPUs
# Gradient accumulation: micro_batch local
# Effective batch: micro_batch × num_gpus × accumulation

8. Memory Fragmentation#

Total free memory: 30 GB
Requested: 5 GB
Result: OOM ❌

Cause#

Detection#

print(torch.cuda.memory_summary())
# "Active memory" ve "Reserved memory" arasındaki gap
# Cached blocks histogram

Çözümler#

1. Expandable segments (PyTorch 2.1+)

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

2. Explicit empty_cache

torch.cuda.empty_cache()  # PyTorch'un cached memory'sini CUDA'ya geri ver

3. Caching disable (debug)

export PYTORCH_NO_CUDA_MEMORY_CACHING=1

4. Better allocator

export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:512

• garbage_collection_threshold
  : %80'i dolduğunda GC tetikle
• max_split_size_mb
  : 512MB'tan büyük block'lar split edilemez (fragmentation azalır)

Pratik öneri#

1. expandable_segments:True
   always
2. max_split_size_mb:512
   yardımcı
3. empty_cache
   her N step (sık değil)
4. Memory snapshot ile fragmentation görsel kontrol

9. Production Memory Budget Planning#

Formül (LLM training)#

Memory ≈ (W × 6) + (G × 2) + A + KV + Overhead

W = parameters (number)
6 = bytes/param (BF16 weight + FP32 master + 8-byte AdamW state, divided)
G = gradient memory (same N, 2 bytes each)
A = activation memory (batch × seq × d × num_layers × ~10-20 bytes)
KV = inference-only KV cache
Overhead = CUDA context (1GB) + PyTorch (0.5GB) + buffers (1-3GB)

Example: 8B Llama 3, batch 8, seq 8192#

• N = 8e9 → W × 6 + G × 2 = 8e9 × 8 = 64 GB
• A ≈ 8 × 8192 × 4096 × 32 × 15 bytes ≈ 130 GB !!

• A_with_checkpointing ≈ 30 GB

Optimization sırası (memory tight)#

1. Mixed precision (BF16) — already
2. Activation checkpointing
3. Gradient accumulation (micro batch küçük)
4. 8-bit optimizer (memory 38% azaltır)
5. FSDP / ZeRO Stage 3 (parameters shard across GPUs)
6. CPU offload (last resort)

Excel template#

10. Production Memory Profiling Checklist#

• Theoretical budget hesaplandı
• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  set
• torch.cuda.empty_cache()
  start'ta
• max_memory_allocated
  track edilir (wandb)
• Activation checkpointing transformer block'larında
• set_to_none=True
  zero_grad'da
• Memory snapshot ilk batch sonrası
• OOM recovery plan: micro_batch küçültme, checkpoint reload

• Memory metric'ler her N step log'la
• Beklenmeyen artış: investigate (memory leak ihtimali)
• Periodic
  empty_cache
  (her 1000 step) eğer fragmentation görüyorsan
• OOM olursa: checkpoint reload + batch yarıya düşür
• Memory snapshot OOM öncesi otomatik kaydet

11. Mini Egzersizler#

1. Memory hesabı: 13B model, BF16, AdamW, batch 4, seq 4096, 40 layer. Toplam memory?
2. Activation checkpointing trade-off: Aktivasyon 50GB → 5GB. Compute overhead ne kadar?
3. OOM diagnosis: 80GB GPU'da training, OOM 3000. step'te. Olası sebepler?
4. Fragmentation detect:

   memory_allocated

   50GB,

   memory_reserved

   75GB, total free 5GB. Fragmentation problemi var mı?
5. Gradient accumulation: GPU 32GB, model batch_size=4'te 30GB. Effective batch=16 nasıl?

Bu Derste Neler Öğrendik?#

Sıradaki Ders#