# Multi-Node Run + Fault-Tolerant Training: 2 Node × 8 H100 NCCL Cluster

> Source: https://sukruyusufkaya.com/learn/fine-tuning-cookbook/ftc-multi-node-fault-tolerant-training
> Updated: 2026-08-14T00:08:43.566Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part IV — Mid-Large Models (13B-70B+) + Distributed Internals
**TLDR:** Cluster training'in gerçeği: node failure'lar olur, NCCL hang olur, checkpoint corrupted olabilir. Cookbook'un fault-tolerant reçetesi: NCCL_TIMEOUT, watchdog, signal handling (SIGUSR1), elastic launcher (torchrun --rdzv_backend=c10d), graceful preemption resume. 70B model 2 günlük training'in 'survival kit'i.