# MoE Mathematical Anatomy: Gating Network, Top-k Routing, Load Balancing — Sparse Activation from Scratch

> Source: https://sukruyusufkaya.com/en/learn/llm-muhendisligi/moe-matematik-anatomi-gating-routing-load-balancing
> Updated: 2026-05-13T13:04:10.292Z
> Category: LLM Mühendisliği
> Module: Module 18: Mixture of Experts (MoE) — Sparse Activation Revolution
**TLDR:** Internal mathematics of MoE: derivation of gating network, top-k routing implementation, expert collapse problem and load balancing loss (Shazeer 2017), auxiliary loss math, capacity factor, drop tokens, FLOP analysis. PyTorch MoE FFN layer implementation from scratch. Expert utilization observations on Turkish data.