MoE Mathematics: Top-K Router + Softmax + Noise + Auxiliary Load-Balancing Loss

1. MoE'nin Çekirdek Fikri#

Mixtral 8×7B:
  - 8 expert per FFN layer
  - Top-K = 2 (her token 2 expert'e gider)
  - Active params per token: 12.9B (out of 46.7B total)
  - Compute: 12.9B model gibi
  - Kapasite: 46.7B model gibi

2. Router Math — Top-K Routing#

Input: x ∈ R^h (token embedding)

# Router weights — küçük linear layer
g(x) = softmax(W_g · x + ε)         # ε = Gaussian noise (load balance için)

# Top-K seçimi
indices = topk(g(x), k=K)           # k=2 Mixtral
weights = g(x)[indices]              # gating weights

# Expert outputs
out = Σ_{i in indices} weights[i] × E_i(x)

# Σ weights = 1 değil (top-K sonrası), bu yüzden:
weights = weights / sum(weights)     # renormalize

3. Expert Capacity Factor#

• capacity_factor = 1.0
  — minimum (overflow olası)
• capacity_factor = 1.25-1.5
  — pratik

• Overflow tokens: Silinir veya residual'a kayar (loss kaybı)

• Padding: Boş slot'lar dolduruluyor (compute kaybı)

4. Auxiliary Load-Balancing Loss#

L_aux = α · N · Σ_i (f_i × P_i)

f_i = fraction of tokens routed to expert i
P_i = average router probability for expert i
N = number of experts
α = 0.01-0.1 (sweet spot 0.01)