# torch.distributed In Depth: DDP, FSDP, ZeRO Stages — Production Distributed Training

> Source: https://sukruyusufkaya.com/en/learn/llm-muhendisligi/torch-distributed-ddp-fsdp-zero-stages
> Updated: 2026-05-13T13:00:25.708Z
> Category: LLM Mühendisliği
> Module: Module 5: PyTorch Engineering — Engineer-Grade
**TLDR:** We covered NCCL fundamentals in 5.4. Now production distributed training stack: DDP gradient bucketing + overlap, FSDP shard strategies (FULL_SHARD, SHARD_GRAD_OP, HYBRID_SHARD), DeepSpeed ZeRO Stage 1/2/3 comparison, hybrid 3D parallelism. Final bridge to Module 17.