# FlashAttention v2/v3 Internals: Tile + Online Softmax + Hopper WGMMA

> Source: https://sukruyusufkaya.com/learn/fine-tuning-cookbook/ftc-flashattention-internals-tile-online-softmax
> Updated: 2026-08-22T07:57:34.185Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part XIII — Custom Kernels & Performance Surgery
**TLDR:** FlashAttention'ın matematiksel kalbi: tile-by-tile attention compute, **online softmax** (incremental running max + sum), backward recomputation strategy. v2 → v3 fark: Hopper WGMMA (warp-group matrix multiply), async memory, FP8 attention. Head-size constraint, deterministic mode, varlen variant.