Vision-Language Models: From CLIP to GPT-4o — Image Encoder + LLM Fusion

Ders Haritası (10 Bölüm)#

1. Pre-VLM era — text-only vs vision models separate
2. CLIP (Radford 2021) — image-text alignment
3. ViT (Dosovitskiy 2020) — vision transformer
4. Image patch embedding — 16x16 patches
5. VLM architecture — encoder + projection + LLM
6. GPT-4V (Eylül 2023) — OpenAI multimodal
7. GPT-4o (Mayıs 2024) — unified multimodal
8. Llama-3.2 Vision (Eylül 2024) — open-source
9. Türkçe multimodal — TR documents OCR/understanding
10. Production deployment — vLLM multimodal

2-5. CLIP + ViT + VLM Architecture#

2.1 CLIP (Radford 2021)#

• Image encoder: ViT or ResNet → image embedding
• Text encoder: transformer → text embedding

Loss: contrastive (positive vs negative pairs)
Image_emb · Text_emb_match >> Image_emb · Text_emb_random

2.2 CLIP impact#

• Zero-shot image classification
• Foundation for ALL modern VLMs
• DALL-E, Stable Diffusion guidance

2.3 ViT (Vision Transformer)#

2.4 Image patch embedding math#

• Patches: 14×14 grid = 196 patches
• Each patch: 16×16×3 = 768 numbers
• Linear projection: 768 → d_model (e.g., 768 for ViT-base)
• • Positional embedding (2D learnable)

2.5 Modern VLM architecture#

Image (224×224×3)
  ↓
[ViT image encoder] → image tokens [N_img, D_vit]
  ↓
[Projection layer] → image tokens [N_img, D_llm]
  ↓                                ↑
[Concatenate with text] ← Text tokens [N_text, D_llm]
  ↓
[LLM transformer]
  ↓
Output text

2.6 Projection layer#

self.image_projection = nn.Linear(d_vit, d_llm)

6-9. GPT-4V, GPT-4o, Llama-3.2 Vision#

6.1 GPT-4V (Eylül 2023)#

6.2 GPT-4o (Mayıs 2024)#

6.3 Llama-3.2 Vision (Eylül 2024)#

• Base Llama-3.1 text model (8B / 70B)
• Vision encoder: CLIP-derived (ViT-H/14)
• Cross-attention layers added to Llama (not just concatenation)
• Pre-trained on 6B image-text pairs

6.4 Llama-3.2 Vision usage#

from transformers import MllamaForConditionalGeneration, AutoProcessor
import torch
from PIL import Image

model = MllamaForConditionalGeneration.from_pretrained(
    "meta-llama/Llama-3.2-11B-Vision-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision-Instruct")

image = Image.open("turkce_belge.png")
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "Bu Türkçe belge ne hakkında? Özetle."},
    ]}
]

inputs = processor(image, processor.apply_chat_template(messages), return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500)
print(processor.decode(outputs[0]))

6.5 Türkçe multimodal use cases#

• Türkçe document OCR + understanding (kimlik, fatura, sözleşme)
• Türkçe chart/graph reading
• KVKK uyumlu visual data processing
• Türk culture image understanding (food, places, art)

6.6 Production deployment#

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.2-11B-Vision-Instruct \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --port 8000

Modül 19 Envanteri (Tamamlandı)#

🏆 MÜFREDAT GENEL TOPLAM#