# SwiGLU Activation: SiLU + GLU = Heart of Modern FFN — From Shazeer 2020 to Llama-3

> Source: https://sukruyusufkaya.com/en/learn/llm-muhendisligi/swiglu-activation-shazeer-2020-llama-3
> Updated: 2026-05-13T13:00:28.436Z
> Category: LLM Mühendisliği
> Module: Module 10: Transformer Block — Anatomy of the Block
**TLDR:** Anatomy of SwiGLU activation function: SiLU (Sigmoid-weighted Linear Unit) base + Gated Linear Unit mechanism. Shazeer 2020 'GLU Variants Improve Transformer'. ReLU/GeLU comparison, why modern models' choice. FFN dimensions (d_ff = 8/3 × d_model Llama-3's choice), parameter math, Llama-3 implementation.