# FSDP + ZeRO: Sharded Training — Rajbhandari 2020'den Llama-3'e Memory Devrim

> Source: https://sukruyusufkaya.com/learn/llm-muhendisligi/fsdp-zero-sharded-training-rajbhandari-2020
> Updated: 2026-08-17T20:23:09.426Z
> Category: LLM Mühendisliği
> Module: Modül 13: Distributed Training — Multi-GPU/Multi-Node
**TLDR:** ZeRO (Zero Redundancy Optimizer, Rajbhandari 2020) — DeepSpeed library: optimizer state, gradients, parameters sharding stage 1/2/3. FSDP (Fully Sharded Data Parallel, PyTorch native) — ZeRO-3 implementation. Llama-3 production: FSDP + activation checkpointing. Memory math: 8B model 1 H100'de eğitilebilir.