# Tensor Parallelism (Megatron): Column-Parallel + Row-Parallel Linear — Splitting the Matrix

> Source: https://sukruyusufkaya.com/en/learn/fine-tuning-cookbook/ftc-tensor-parallelism-megatron-column-row
> Updated: 2026-05-14T14:42:52.492Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part IV — Mid-Large Models (13B-70B+) + Distributed Internals
**TLDR:** Megatron-LM (NVIDIA) Tensor Parallel: matrix split *within itself* across GPUs. Column-parallel linear (output channels split), row-parallel (input channels), all-reduce/gather pattern. TP=2 vs TP=4 on 8×H100. FSDP+TP = 2D parallelism.