Mixture of Experts (MoE): Sparse Activation Revolution — From Mixtral 8x7B to DeepSeek-V3

Ders Haritası (10 Bölüm)#

1. Dense vs sparse model — niye MoE
2. Shazeer 2017 — original MoE paper
3. Top-k gating — routing mechanism
4. Expert FFN — each expert is small FFN
5. Routing math — softmax → top-k → weighted sum
6. Auxiliary loss — load balancing
7. Mixtral 8x7B (Ocak 2024) — open-source breakthrough
8. DeepSeek-V3 671B — frontier scale MoE
9. MoE inference — vLLM, dense-equivalent FLOPs
10. Türkçe için — pratik implications

2-6. MoE Math#

2.1 Dense FFN recap#

FFN(x) = down(silu(gate(x)) * up(x))

2.2 MoE intuition#

MoE(x) = Σ_i gate(x)_i × expert_i(x)

2.3 Top-k sparse activation#

weights = softmax(linear(x))  # [B, S, N]
top_k_weights, top_k_indices = topk(weights, k=2)  # only k experts
out = Σ_{i in top_k} top_k_weights_i × expert_i(x)

2.4 Per-token routing#

• 'kapı' → experts 1, 3
• 'yı' → experts 2, 5
• 'çal' → experts 1, 7

2.5 Load balancing problem#

L_aux = N × Σ_i fraction_i × probability_i

• fraction_i: i. expert'in toplam routing fraction'ı
• probability_i: i. expert'in ortalama gate probability'si

2.6 Implementation outline#

class MoELayer(nn.Module):
    def __init__(self, d_model, n_experts, k=2):
        super().__init__()
        self.n_experts = n_experts
        self.k = k
        self.gate = nn.Linear(d_model, n_experts)
        self.experts = nn.ModuleList([
            SwiGLUFFN(d_model) for _ in range(n_experts)
        ])
    
    def forward(self, x):
        # x: [batch, seq, d_model]
        gates = F.softmax(self.gate(x), dim=-1)
        top_k_gates, top_k_idx = gates.topk(self.k, dim=-1)
        
        # Dispatch to top-k experts (simplified)
        out = torch.zeros_like(x)
        for i in range(self.n_experts):
            mask = (top_k_idx == i).any(dim=-1)
            if mask.any():
                expert_out = self.experts[i](x[mask])
                # Weight by gate probability
                gate_weight = (top_k_gates * (top_k_idx == i).float()).sum(-1)[mask]
                out[mask] += expert_out * gate_weight.unsqueeze(-1)
        return out

7-10. Mixtral + DeepSeek-V3#

7.1 Mixtral 8x7B (Mistral AI, Ocak 2024)#

• 8 experts per FFN layer
• top-2 routing
• 47B total params, 13B active per token
• Quality: comparable Llama-2-70B (4x less compute)

7.2 Mixtral architectural#

for each transformer layer:
  Attention (dense)
  MoE FFN (8 experts, top-2)
  RMSNorm + residual

7.3 DeepSeek-V3 (Aralık 2024)#

• 256 experts per FFN layer
• top-8 routing
• • 1 shared expert (always active)
• 671B total, 37B active per token

7.4 DeepSeek-V3 quality#

• MMLU: 88.5%
• MATH: 90.2%
• Coding (HumanEval): 82.3%
• Better than Llama-3-70B, comparable GPT-4o
• Training cost: $5.6M (extraordinarily efficient)

7.5 MoE inference#

• Mixtral 47B model size: ~90 GB (bf16) → 2x A100 80GB
• Inference compute equivalent to 13B dense → much faster than 47B dense

7.6 Türkçe için MoE#

7.7 Future MoE#

• Better specialization
• More flexible routing
• Frontier model standard

Modül 18 Envanteri (Tamamlandı)#