LLaVA-1.5 / 1.6 / OneVision: 2-Stage Training + Projector Pretrain + Instruction Tune

1. LLaVA 2-Stage Training#

Stage 1: Projector Pretrain
  - Frozen: Vision encoder, LLM
  - Trainable: Projector (MLP) only
  - Data: 558K LAION-CC-SBU image-caption pairs
  - Format: <image>{caption}
  - Süre: ~12 saat 8×A100 80GB
  - Amaç: Image embeddings'i LLM embedding space'iyle align et

Stage 2: Visual Instruction Tune
  - Frozen: Vision encoder (genelde)
  - Trainable: LLM (full or LoRA) + Projector
  - Data: 150K-665K visual instruction pairs (LLaVA-Instruct + custom)
  - Format: <image>\nUser: question\nAssistant: answer
  - Süre: ~10 saat 8×A100 80GB (Vicuna 13B)
  - Amaç: Multimodal instruction following

2. Freeze Strategy Ablation (Llava-1.6 Mistral 7B, RTX 4090 QLoRA)#