Capstone Modül 8: Quadratic Attention'a Alternatifler — Linear Attention, RetNet, Mamba (State Space Models)

Capstone Akışı (8 Aşama)#

1. Quadratic problem — fundamental limit recap
2. Linear Attention (Katharopoulos 2020) — kernel trick
3. RetNet (Sun 2023) — Microsoft retention mechanism
4. State Space Models — Gu Dao 2023, S4 → Mamba
5. Mamba detail — selective scan, hardware-aware
6. Empirical comparison — quality vs efficiency
7. Hybrid models — Jamba (attention + Mamba)
8. 2026 forecast — gelecek mimariler

1. Quadratic Problem Recap#

1.1 Standard attention complexity#

Memory:  O(seq²)
Compute: O(seq² × d)

• Memory (without FA): 32 GB per head
• Compute: 6.7 × 10^13 FLOP

1.2 FlashAttention'ın çözdüğü#

1.3 Sub-quadratic ne demek#

• Linear: O(seq × d)
• Log-linear: O(seq × log(seq) × d)
• Sparse: O(k × seq × d) with k attention pattern density

2. Linear Attention — Kernel Trick#

2.1 Katharopoulos 2020#

Attention(Q, K, V) = φ(Q) (φ(K)^T V) / (φ(Q) φ(K)^T 1)

• φ(K)^T V önce hesaplanır: [d, d] matrix
• Sonra φ(Q) ile çarpılır: [seq, d] @ [d, d] = [seq, d]
• Toplam compute: O(seq × d²) — linear in seq!

2.2 Recurrent form#

S_t = S_{t-1} + φ(k_t) v_t^T   # state update
q_t = φ(q_t)
output_t = q_t S_t / (q_t Σ φ(k_i))

2.3 φ kernel choices#

• φ(x) = elu(x) + 1 — Katharopoulos default
• φ(x) = exp(x) — closer to softmax
• φ(x) = x² — Performer FAVOR+

2.4 Quality concern#

4-5. Mamba — Selective State Space Models#

4.1 State Space Models (SSM) tarihçesi#

• S4 (Gu 2021): efficient long-range dependencies
• S5: simpler implementation
• Mamba (Gu Dao 2023): selective + hardware-aware

4.2 Mamba core idea#

h_t = A h_{t-1} + B x_t
y_t = C h_t + D x_t

4.3 Selective scan algorithm#

Memory:  O(seq × d_state)
Compute: O(seq × d × d_state)

4.4 Mamba vs Transformer scaling#

• Same params, same training tokens
• Small models (~3B): comparable quality
• Long context (16K, 64K): Mamba dramatic better
• Mamba inference 5x faster (no KV cache!)

4.5 No KV cache!#

4.6 Limitations#

4.7 Production status (2026)#

• Falcon-Mamba 7B (Technology Innovation Institute)
• Mistral-NeMo Mamba variants
• Codestral Mamba (Mistral)
• Hybrid: Jamba (AI21 Labs) — Mamba + Attention layers alternating

7-8. Hibrit Modeller + 2026 Forecast#

7.1 Jamba (AI21 Labs 2024)#

[Mamba × 4] → [Attention × 1] → [Mamba × 4] → [Attention × 1] → ...

• Memory: %30 of pure transformer
• Quality: comparable Llama-2-70B
• Long context (256K): native, KV cache çoğunlukla Mamba'da

7.2 2026 trend: Mamba/SSM mainstreaming#

• Falcon-Mamba production-ready
• MoE + Mamba combo trials
• Multi-modal Mamba (vision + text)

7.3 Transformer'ın savunması#

• ICL (in-context learning) Mamba'dan iyi
• Mature ecosystem (PyTorch, vLLM, HF integration)
• Pre-training data abundance

7.4 Pragmatic advice (2026)#

• General-purpose LLM: Transformer (Llama-3, GPT-4)
• Long context (100K+): Hybrid (Jamba) veya pure Mamba
• Real-time inference: Mamba (no KV cache)
• Research: explore both

7.5 Türkçe için#

Modül 8 Envanteri (Tamamlandı)#