SwiGLU Activation: SiLU + GLU = Modern FFN'in Kalbi — Shazeer 2020'den Llama-3'e

Ders Haritası (10 Bölüm)#

1. FFN'in role — niye attention sonrası FFN
2. Activation function evrim — ReLU → GeLU → SwiGLU
3. SiLU (Swish) — formül + intuition
4. GLU (Gated Linear Unit) — Dauphin 2017
5. SwiGLU formula — SiLU + GLU combination
6. Shazeer 2020 — 9 variant karşılaştırma
7. Llama-3 FFN architecture — d_ff = 8/3 × d_model
8. Parameter count — 3 matrix vs 2 matrix
9. Implementation — production-grade PyTorch
10. Empirical performance — quality vs compute

1-5. FFN + Activation Function Evrim#

1.1 FFN'in role#

FFN(x) = activation(x W_1) W_2

• Attention contextualized representation'ı transform et
• Non-linear transformation (model capacity)
• Token-wise (parallel possible)

1.2 ReLU (Vaswani 2017)#

FFN(x) = max(0, xW_1 + b_1) W_2 + b_2

1.3 GeLU (BERT 2018, GPT-2)#

GeLU(x) = x × Φ(x)        # Φ = standard normal CDF
        ≈ 0.5x(1 + tanh(sqrt(2/π)(x + 0.044715x³)))

1.4 SiLU / Swish (Ramachandran 2017, Hendrycks 2016)#

SiLU(x) = x × σ(x)        # σ = sigmoid

1.5 GLU (Gated Linear Unit, Dauphin 2017)#

GLU(x) = σ(x W_v) ⊙ (x W_u)

• W_v: 'gate' (sigmoid filter)
• W_u: 'value' (information to pass)
• Element-wise product: gate × value

1.6 SwiGLU (Shazeer 2020)#

SwiGLU(x) = SiLU(x W_g) ⊙ (x W_u)
          = (x W_g × σ(x W_g)) ⊙ (x W_u)

1.7 Empirical: Shazeer 2020#

• ReLU: baseline
• GeLU: +0.3 PPL
• Swish: +0.4 PPL
• GLU: +0.8 PPL
• ReGLU: +1.0 PPL
• GeGLU: +1.3 PPL
• SwiGLU: +1.5 PPL (best)

7-9. Llama-3 FFN Architecture#

7.1 Llama-3 FFN formula#

FFN(x) = (SiLU(x W_gate) ⊙ (x W_up)) W_down

• W_gate: d_model → d_ff
• W_up: d_model → d_ff
• W_down: d_ff → d_model

7.2 d_ff dimension choice#

d_ff = 8/3 × d_model ≈ 2.67 × d_model

7.3 Parameter accounting#

• ReLU/GeLU FFN: 2 × d_model × d_ff = 2 × 4096 × 16384 = 134M (d_ff=4×d_model)
• SwiGLU FFN: 3 × d_model × d_ff = 3 × 4096 × 11008 = 135M (d_ff=8/3×d_model)

7.4 Llama-3 production implementation#

class LlamaFFN(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.gate_proj = nn.Linear(d_model, d_ff, bias=False)
        self.up_proj = nn.Linear(d_model, d_ff, bias=False)
        self.down_proj = nn.Linear(d_ff, d_model, bias=False)
    
    def forward(self, x):
        gate = F.silu(self.gate_proj(x))
        up = self.up_proj(x)
        return self.down_proj(gate * up)

7.5 SwiGLU memory footprint#

• d_ff = 11008, batch=32, seq=2048: 32 × 2048 × 11008 × 2 byte (bf16) = 1.4 GB per layer
• 32 layers: 45 GB total intermediate (just FFN)

Sıradaki Ders: Transformer Block Bütünü#