AdamW + Learning Rate Schedule: Modern LLM Optimization'ın Matematik Anatomisi

Ders Haritası (10 Bölüm)#

1. SGD'den Adam'a evrim — momentum, adaptive lr
2. Adam (Kingma 2014) — first moment + second moment
3. L2 vs weight decay — niye farklı
4. AdamW (Loshchilov 2019) — decoupling weight decay
5. Hyperparameters — β1, β2, ε, weight_decay seçimi
6. Learning rate schedule — warmup + cosine decay
7. Niye warmup — adaptive lr stabilization
8. Gradient clipping — explosion prevention
9. Mixed precision considerations — fp32 master weights
10. Llama-3 hyperparameters — production values

1-4. SGD → Adam → AdamW#

1.1 SGD#

θ_{t+1} = θ_t - lr × ∇L(θ_t)

1.2 SGD + momentum#

m_{t+1} = β × m_t + ∇L(θ_t)
θ_{t+1} = θ_t - lr × m_{t+1}

1.3 Adam (Kingma 2014)#

• First moment (mean): m_t = β1 × m_{t-1} + (1-β1) × g_t
• Second moment (variance): v_t = β2 × v_{t-1} + (1-β2) × g_t²

• m_hat = m_t / (1 - β1^t)
• v_hat = v_t / (1 - β2^t)

θ_{t+1} = θ_t - lr × m_hat / (sqrt(v_hat) + ε)

1.4 Adam + L2 weight decay#

L_total = L_task + (λ/2) × ||θ||²

m_t = β1 × m_{t-1} + (1-β1) × (∇L_task + λθ_t)

1.5 AdamW (Loshchilov 2019)#

# Gradient update (no weight decay in gradient)
m_t = β1 × m_{t-1} + (1-β1) × g_t
v_t = β2 × v_{t-1} + (1-β2) × g_t²
m_hat = m_t / (1 - β1^t)
v_hat = v_t / (1 - β2^t)

# Apply update + decoupled weight decay
θ_{t+1} = θ_t - lr × (m_hat / (sqrt(v_hat) + ε) + weight_decay × θ_t)

1.6 Empirical improvement#

• AdamW: better generalization
• Lower validation loss
• Modern LLM models all AdamW

1.7 PyTorch AdamW#

import torch.optim as optim

optimizer = optim.AdamW(
    model.parameters(),
    lr=3e-4,
    betas=(0.9, 0.95),    # β1, β2
    eps=1e-5,
    weight_decay=0.1,
)

6-9. Learning Rate Schedule#

6.1 Niye constant lr yetmez#

• Erken: büyük updates (model far from optimum)
• Geç: küçük updates (fine-tune around optimum)

6.2 Linear warmup#

lr(t) = max_lr × min(1, t / warmup_steps)

• AdamW second moment v_t başlangıçta unstable
• Çok küçük v_t → ratio çok büyük → updates patlama
• Warmup ile v_t stabilize

6.3 Cosine decay#

lr(t) = lr_min + 0.5 × (max_lr - lr_min) × (1 + cos(π × (t - warmup) / (total - warmup)))

6.4 Linear decay#

6.5 Llama-3 schedule#

• warmup: 2000 steps (linear)
• decay: cosine, max 3e-4 → min 3e-5 (10x)
• total: 1.4M steps

6.6 Gradient clipping#

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

6.7 Mixed precision#

• Forward + backward: bf16 (BFloat16)
• Optimizer state: fp32 (full precision)
• Master weights: fp32
• Gradient: bf16 → fp32 conversion

6.8 Production checklist#

from torch.optim.lr_scheduler import LambdaLR
import math

def cosine_schedule_with_warmup(step, warmup_steps, total_steps, min_ratio=0.1):
    if step < warmup_steps:
        return step / warmup_steps
    progress = (step - warmup_steps) / (total_steps - warmup_steps)
    return min_ratio + 0.5 * (1 - min_ratio) * (1 + math.cos(math.pi * progress))

scheduler = LambdaLR(optimizer, lambda step: cosine_schedule_with_warmup(step, 2000, 1_400_000))

Sıradaki Ders: Mini Llama-3 Training Capstone#