Quantization (GPTQ/AWQ/GGUF) + Final Capstone: Turkish ChatGPT Clone in Production

Final Capstone Akışı (10 Aşama)#

1. Sistem mimarisi — full-stack overview
2. Quantization options — GPTQ vs AWQ vs GGUF
3. AWQ quantize — Türkçe Llama-3-8B → 4-bit
4. vLLM backend — quantized model serving
5. Next.js frontend — chat UI + streaming
6. API integration — frontend ↔ vLLM
7. Authentication + rate limiting
8. Production deploy — Vercel frontend + GPU cloud backend
9. Monitoring + analytics
10. Final review — müfredatın 5 capstone'u + ne kazandın

2-3. Quantization Formats#

2.1 Quantization niye#

2.2 GPTQ (Frantar 2022)#

pip install auto-gptq

2.3 AWQ (Lin 2023)#

pip install autoawq

2.4 GGUF (llama.cpp)#

2.5 Quantization quality comparison#

• BF16: 8.2
• AWQ 4-bit: 8.5 (+%4)
• GPTQ 4-bit: 8.7 (+%6)
• GGUF Q4_K_M: 8.4 (+%2.5)

2.6 AWQ Türkçe Llama-3 quantize#

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "sukruyusufkaya/llama-3-8b-tr-instruct"
quant_path = "sukruyusufkaya/llama-3-8b-tr-instruct-awq"

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Calibration data: Türkçe samples
calibration_data = ["Türkçe örnek metin 1...", "Türkçe örnek metin 2...", ...]

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

model.quantize(tokenizer, quant_config=quant_config, calib_data=calibration_data)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

5-8. Full Stack#

5.1 vLLM backend#

# Quantized model serve
python -m vllm.entrypoints.openai.api_server \
  --model sukruyusufkaya/llama-3-8b-tr-instruct-awq \
  --quantization awq \
  --gpu-memory-utilization 0.9 \
  --port 8000

5.2 Next.js frontend#

// app/api/chat/route.ts
export async function POST(req: Request) {
  const { messages } = await req.json();
  
  const response = await fetch('http://vllm-backend:8000/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'sukruyusufkaya/llama-3-8b-tr-instruct-awq',
      messages,
      stream: true,
      temperature: 0.7,
      max_tokens: 500,
    }),
  });
  
  // Stream response back to client
  return new Response(response.body, {
    headers: { 'Content-Type': 'text/event-stream' },
  });
}

5.3 Chat UI#

// components/Chat.tsx
import { useState } from 'react';

export default function Chat() {
  const [messages, setMessages] = useState([]);
  const [input, setInput] = useState('');
  
  const send = async () => {
    const newMessages = [...messages, { role: 'user', content: input }];
    setMessages(newMessages);
    setInput('');
    
    const res = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ messages: newMessages }),
    });
    
    // Stream handling...
    const reader = res.body!.getReader();
    let assistant = '';
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      assistant += new TextDecoder().decode(value);
      setMessages([...newMessages, { role: 'assistant', content: assistant }]);
    }
  };
  
  return (
    <div>
      {messages.map((m, i) => <p key={i}><b>{m.role}:</b> {m.content}</p>)}
      <input value={input} onChange={e => setInput(e.target.value)} />
      <button onClick={send}>Gönder</button>
    </div>
  );
}

5.4 Auth + rate limiting#

• NextAuth for user authentication
• Rate limit per user per minute (e.g., 30 messages)
• Optional: Stripe billing tiers

5.5 Production deploy#

• Frontend: Vercel (Next.js)
• Backend: GPU cloud (Runpod, Vast.ai, AWS EC2 G5)
• Domain: ai-asistan.sukruyusufkaya.com

🏆 Müfredat Tamamlandı — Genel Toplam#

Tüm Modüller#

5 Capstone Artifact#

1. TurkTokenizer-tr 32K BPE (Modül 6)
2. Türkçe Semantic Search Mini-RAG (Modül 7)
3. Mini Llama-3 100M Param Türkçe Pretrain (Modül 11)
4. Türkçe Llama-3-8B-Instruct Fine-Tune (Modül 14)
5. Türkçe ChatGPT Klonu Production (Modül 16)

Türkiye'nin En Kapsamlı LLM Mühendisliği Müfredatı#