# Tensor Parallelism (Megatron): Column-Parallel + Row-Parallel Linear — Matrix'i Böl

> Source: https://sukruyusufkaya.com/learn/fine-tuning-cookbook/ftc-tensor-parallelism-megatron-column-row
> Updated: 2026-08-23T09:59:55.145Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part IV — Mid-Large Models (13B-70B+) + Distributed Internals
**TLDR:** Megatron-LM (NVIDIA) Tensor Parallel: bir matrix'in **kendi** içinde GPU'lar arasında bölünmesi. Column-parallel linear (output channels split), row-parallel linear (input channels split), all-reduce/gather pattern. 8×H100'de TP=2 vs TP=4 karar matrisi. FSDP+TP combine = 2D parallelism.