Sparse Attention

Sparse attention was developed to reduce the quadratic complexity of standard self-attention. In long-context tasks, letting every token attend to every other token becomes expensive, so only selected attention patterns are used. This is important in long-document modeling, genomic data, and large-context language models. It creates new trade-offs between computational efficiency and representational richness.