Tokenizer Evaluation: Fertility, Compression Ratio, Downstream Impact, and Information-Theoretic Metrics

Ders Haritası (14 Bölüm)#

1. Niye ölç — tokenizer kalitesinin maliyet ve performans etkisi
2. Fertility — token/word, derin matematik ve Türkçe pratik
3. Compression ratio — bytes/token, information theory bağlantısı
4. OOV rate — out-of-vocabulary metrikleri
5. Bits-per-character (BPC) — bilgi teorik standart
6. Vocab coverage — domain ve dil için kapsam analizi
7. Perplexity'ye etki — tokenization → language model quality
8. Downstream task impact — NER, QA, classification accuracy
9. Cross-lingual fertility — adil karşılaştırma protokolleri
10. Token efficiency rate — semantic information per token
11. A/B testing — tokenizer karşılaştırma protokolü
12. Tokenization tax — Türkçe maliyet kanıtı
13. Çoklu metrik dashboard — capstone için evaluation framework
14. Edge cases — outlier domains, code, math, emoji, multilingual

1. Niye Ölç — Tokenizer'ın Gizli ROI'si#

1.1 Ölçmemenin maliyeti#

• Inference cost: her API call'da token sayısı = direkt $ maliyet
• Latency: daha az token = daha az model forward pass = daha hızlı yanıt
• Quality: kötü tokenization → semantic information loss → daha kötü reasoning
• Context window: daha verimli tokenizer = aynı context'te daha çok bilgi

1.2 Türkçe-spesifik etki (production data)#

1.3 Quality etkisi (BERT-base-Turkish vs mBERT)#

• mBERT (110K multilingual): F1 = 0.881
• BERT-base-Turkish (32K TR-only): F1 = 0.912 (+3.5%)

1.4 Ölçüm disiplinin akademik temeli#

• Bostrom & Durrett 2020: 'Byte Pair Encoding is Suboptimal for Language Model Pretraining'
• Rust et al. 2020: 'How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models'
• Park & Park 2021: 'An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks'
• Petrov et al. 2023: 'Language Model Tokenizers Introduce Unfairness Between Languages'

1.5 Ölçüm felsefesi#

• General LLM: fertility + perplexity + downstream avg
• Classification: fertility + class-balanced accuracy
• NER: fertility + entity boundary F1
• Code: byte coverage + python/JS-specific fertility
• Math: digit/number tokenization patterns

2. Fertility — Token/Word'ün Derin Matematiği#

2.1 Tanım#

fertility = total_tokens / total_words

2.2 Hesaplama (basit)#

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

text = "İstanbul Boğazı'nda balıkçı tekneleri sallanıyor."
words = text.split()  # whitespace-based
tokens = tokenizer.tokenize(text)

fertility = len(tokens) / len(words)
print(f"Fertility: {fertility:.2f}")

2.3 Word tanımı — kritik karar#

• Whitespace split: `text.split()` — en basit, en yaygın
• Linguistic word: morfolojik analiz (Türkçe için Zemberek/Morfessor)
• Punctuation-aware: punctuation ayrı/değil

2.4 Türkçe morfolojik komplikasyon#

morph_fertility = total_tokens / total_morphemes

2.5 Fertility heterogeneity#

• Common words ('ev', 'kitap'): fertility ~1.0
• Rare words ('elektromanyetik'): fertility 3-5
• Numbers ('2026'): fertility 1-2
• Proper nouns ('İstanbul'): fertility ~1.0 (frequent)
• URLs, emails: fertility 5-10 (high)

2.6 Türkçe gerçek sayılar#

2.7 Domain etkisi#

• Wikipedia: baseline
• News: ~%10 daha düşük (well-structured)
• Literature: ~%5 daha yüksek (rich vocab)
• Twitter/sosyal medya: ~%15 daha yüksek (slang, typos)
• Legal: ~%20 daha yüksek (formal terminology)
• Code: aşırı domain-specific (Python/JS BPE'siz çok yüksek)

3. Compression Ratio — Bytes/Token, Information Theory#

3.1 Tanım#

compression_ratio = total_bytes / total_tokens

3.2 Hesaplama#

text = "İstanbul Boğazı'nda balıkçı tekneleri sallanıyor."
bytes_count = len(text.encode("utf-8"))
tokens = tokenizer.tokenize(text)
ratio = bytes_count / len(tokens)
print(f"Bytes per token: {ratio:.2f}")

3.3 Yüksek mi düşük mü iyi?#

• Çok yüksek (>5 bytes/token) — vocab gereksiz büyük, sub-optimal merge
• Çok düşük (<2 bytes/token) — vocab küçük, char-level'a yakın
• Sweet spot: 3.5-5 bytes/token Türkçe için

3.4 Information theory bağlantısı#

3.5 Karşılaştırma (Türkçe 1 MB metin)#

3.6 Compression vs fertility ilişkisi#

• Yüksek compression → düşük fertility (genelde)
• Aynı şeyi farklı boyutlardan ölçüyorlar

• Fertility: linguistic unit (word) bazlı
• Compression: information unit (byte) bazlı

4. OOV Rate — Out-of-Vocabulary Ölçümü#

4.1 Klasik OOV (vocab kelime-bazlı)#

4.2 Subword OOV#

• 1 parça: 'in-vocab'
• 2+ parça: 'split'
• Word-level OOV rate = % words split into 2+ tokens

def oov_rate(text, tokenizer):
    words = text.split()
    split_words = 0
    for w in words:
        if len(tokenizer.tokenize(w)) > 1:
            split_words += 1
    return split_words / len(words)

4.3 Türkçe OOV gerçek#

4.4 Hard OOV vs soft OOV#

• Hard OOV: `` token (tokenizer'da byte fallback yoksa). Modern modern model'lerde nadir.
• Soft OOV: word birkaç subword'e bölünüyor. Yaygın, kabul edilebilir.

4.5 OOV vs fertility ilişkisi#

fertility ≥ 1 + (OOV rate × avg_subwords_per_oov)

fertility ≥ 1 + 0.30 × 2 = 1.6

4.6 Domain-specific OOV#

5. Bits-Per-Character (BPC) — Information-Theoretic Standard#

5.1 Tanım#

BPC = -log2(P(text)) / len(text_in_chars)

5.2 Tokenizer-aware BPC#

P(text) = ∏ over tokens t: P(t | history)
-log2(P(text)) = Σ -log2(P(t | history))

BPC = total_token_logprob_loss / total_chars

5.3 Niye karakter bazlı normalize ediliyor#

• Tokenizer A: 100 token, total loss 50
• Tokenizer B: 50 token, total loss 30
• Char sayısı: 200 her ikisi için

5.4 Production model BPC değerleri#

5.5 Tokenizer'ın BPC'ye etkisi#

5.6 BPC'yi hesaplamak#

import torch
import torch.nn.functional as F

def compute_bpc(model, tokenizer, text):
    tokens = tokenizer.encode(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(tokens, labels=tokens)
    nll = outputs.loss.item() * (tokens.size(1) - 1)  # total NLL
    total_chars = len(text)
    bpc = nll / total_chars / math.log(2)  # nats → bits
    return bpc

5.7 BPC vs Perplexity#

• Perplexity = exp(loss_per_token) — token-bazlı, fertility'ye duyarlı
• BPC = loss_per_char / log(2) — char-bazlı, fertility-agnostic

6. Vocab Coverage — Kapsam Analizi#

6.1 Domain coverage#

test_corpora = {
    "wikipedia": load_wiki_tr(),
    "news": load_news_tr(),
    "legal": load_legal_tr(),       # Yargıtay decisions
    "medical": load_medical_tr(),    # ilaç prospektüsleri
    "code": load_code_comments_tr(),
    "social": load_twitter_tr(),
}

for domain, corpus in test_corpora.items():
    print(f"{domain}: fertility={fertility(corpus, tokenizer):.2f}")

6.2 Sample analiz (gerçek)#

• Wikipedia: 1.20
• News: 1.18
• Legal: 2.10 (out of domain)
• Medical: 2.35
• Code: 2.80
• Social: 1.65

6.3 Çözüm: domain-augmented training#

6.4 Character coverage#

def char_coverage(text, tokenizer):
    encoded = tokenizer.encode(text)
    decoded = tokenizer.decode(encoded)
    matches = sum(1 for a, b in zip(text, decoded) if a == b)
    return matches / len(text)

6.5 Multilingual coverage#

7. Perplexity'ye Etki — Tokenizer ↔ LM Quality#

7.1 Aynı model, farklı tokenizer#

• Tokenizer A: 32K vocab → embedding 32K × 4096 = 131M params
• Tokenizer B: 100K vocab → embedding 100K × 4096 = 409M params

7.2 Empirical findings (Llama-3 paper)#

• 32K: PPL 6.8
• 128K: PPL 6.2 (-9%)

7.3 Vocab vs depth trade-off#

• Küçük vocab + derin model: morfolojik dilleri öğrenir, vocabüleri 'ararken' parametre harcar
• Büyük vocab + sığ model: kelimeleri ezberler, reasoning sığ

7.4 Türkçe için optimum (theoretical)#

7.5 BPC ile perplexity link#

8. Downstream Task Impact — Gerçek Dünya#

8.1 NER (Named Entity Recognition) Türkçe#

8.2 Sentiment classification#

8.3 Question Answering (Türkçe SQuAD)#

8.4 Math reasoning#

• Per-digit tokenization (cl100k): %85 accuracy
• Group-of-3 (yeni o200k): %91 accuracy
• Per-char: %94 accuracy

8.5 Code generation#

• Python-specific tokenizer: %X higher HumanEval pass rate
• General tokenizer: indent + bracket fertility yüksek → less efficient

8.6 Üzerinden geçen pattern#

• Avantaj: tuned tokenizer task'ı daha iyi yakalar
• Dezavantaj: pre-trained model'ler genel-amaç tokenizer kullanır, swap olunmaz

9. Cross-Lingual Fertility — Adil Karşılaştırma#

9.1 Petrov 2023: 'Tokenizer Unfairness'#

9.2 Adil karşılaştırma protokolü#

from datasets import load_dataset
ds = load_dataset("facebook/flores", "tur_Latn")
for sentence in ds["devtest"]:
    en_text = sentence["sentence_eng_Latn"]
    tr_text = sentence["sentence_tur_Latn"]
    en_tokens = tokenizer.tokenize(en_text)
    tr_tokens = tokenizer.tokenize(tr_text)
    ratio = len(tr_tokens) / len(en_tokens)
    print(f"TR/EN ratio: {ratio:.2f}")

9.3 GPT-4o tokenizer FLORES ratios#

9.4 Niye var#

• Pre-training corpus İngilizce-dominant (%60+)
• Tokenizer training corpus İngilizce-dominant → İngilizce'ye iyi tune
• Türkçe under-represented → fertility yüksek

9.5 Çözüm yolları#

1. Balanced corpus: per-language quota
2. Bigger vocab: o200k → o500k
3. Per-language adapter: dile özel embedding
4. TR-only tokenizer fine-tune: tamamen Türkçe-tuned (downstream re-train gerekir)

9.6 Adalet metriği#

10. Token Efficiency Rate — Semantik Bilgi/Token#

10.1 Tanım#

10.2 Lossy proxy: information per token#

bits_per_token = -log2(P(token | history))

10.3 Yüksek yoğunluk vs düşük yoğunluk tokenlar#

• High entropy (rare/surprising): 'electromagnetic' tek token
• Low entropy (predictable): 'the', ' the', '##in' (continuation)

10.4 Aynı bilgi farklı tokenizer'larda#

• 7 kelime, 12 token (GPT-4o)
• Information content: ~25 bits (English LM proxy)
• Bits/token: ~2.1

10.5 Efficiency vs compression trade-off#

• Çok büyük vocab → her token rare → yüksek entropy → ama parameter cost yüksek
• Çok küçük vocab → her token frequent → düşük entropy → fertility yüksek

11. A/B Testing — Tokenizer Karşılaştırma Protokolü#

11.1 Fair comparison challenges#

• Model embedding tokenizer-specific
• Pre-training corpus farklı olabilir
• Hyperparameter optimization tokenizer'a bağlı

11.2 Controlled study tasarımı#

1. Same architecture: aynı transformer (GPT-2 small)
2. Same total params: vocab × dim sabit
3. Same training corpus: identical pre-training data
4. Same training steps: aynı compute
5. Same downstream eval: identical test sets

11.3 Metric panel#

• Pre-training: BPC, validation loss
• Downstream: 5-10 task average
• Efficiency: throughput, memory
• Coverage: domain analysis

11.4 Statistical significance#

11.5 Production A/B testing#

• Latency (p50, p99)
• User satisfaction (thumbs up rate)
• Conversation length distribution
• Cost per conversation

11.6 Capstone evaluation framework#

1. Baseline: Llama-3 default tokenizer
2. Variant: TurkBPE 32K (Türkçe-tuned)
3. Metrics: fertility, BPC, downstream NER+QA+sentiment
4. Output: detaylı report

12. Tokenization Tax — Türkçe Maliyet Kanıtı#

12.1 Tax kavramı#

12.2 Hesaplama#

tax_rate = (TR_tokens / EN_tokens) - 1

12.3 Yıllık tax (gerçek hesap)#

• TR cost: 10M × 500 × $2.5/1M =$12,500/ay = $150K/yıl
• EN equivalent: 10M × 280 × $2.5/1M =$7,000/ay = $84K/yıl
• Tax: $66K/yıl

12.4 Tax azaltma stratejileri#

1. Better tokenizer: o200k Türkçe %20 indirim (geçtiğimiz nesil cl100k'dan)
2. TR-tuned model: Trendyol-LLM tax 0.20 civarında
3. Self-host: kendi modelin Türkçe-tuned vocab → tax sıfır (ama infra cost var)
4. Hybrid routing: simple queries TR-tuned local, complex queries OpenAI

12.5 Long-term outlook#

• o200k → o500k (önümüzdeki 2 yıl)
• Per-language vocab quota tasarımları artıyor
• Open-source TR modellerin yetkinliği artıyor

13. Çoklu Metrik Dashboard — Capstone Evaluation#

13.1 Standart evaluation script#

def evaluate_tokenizer(tokenizer, test_corpus, languages=None):
    metrics = {}
    
    # 1. Fertility
    words = sum(len(line.split()) for line in test_corpus)
    tokens = sum(len(tokenizer.tokenize(line)) for line in test_corpus)
    metrics["fertility"] = tokens / words
    
    # 2. Compression
    bytes_total = sum(len(line.encode("utf-8")) for line in test_corpus)
    metrics["bytes_per_token"] = bytes_total / tokens
    
    # 3. OOV rate
    split_count = sum(
        1 for line in test_corpus for word in line.split()
        if len(tokenizer.tokenize(word)) > 1
    )
    metrics["oov_rate"] = split_count / words
    
    # 4. Vocab utilization
    used_tokens = set()
    for line in test_corpus:
        used_tokens.update(tokenizer.encode(line))
    metrics["vocab_used"] = len(used_tokens) / tokenizer.vocab_size
    
    # 5. Avg token length
    metrics["avg_token_length_bytes"] = bytes_total / tokens
    
    return metrics

13.2 Comparison table#

def compare_tokenizers(tokenizers_dict, test_corpora_dict):
    results = {}
    for tname, tok in tokenizers_dict.items():
        for dname, corpus in test_corpora_dict.items():
            key = f"{tname}_{dname}"
            results[key] = evaluate_tokenizer(tok, corpus)
    return pd.DataFrame(results).T

13.3 Visualizing#

• Heatmap: tokenizer × domain fertility
• Bar chart: cross-lingual fertility distribution
• Distribution: per-word fertility histogram (mean, percentiles)
• Scatter: vocab size vs fertility curve

13.4 Reporting template#

• Executive summary (3 satır)
• Method (training corpus, hyperparams)
• Metrics (fertility, BPC, OOV, downstream)
• Comparison (baseline + 2-3 alternative)
• Limitations (domain coverage, multilingual)
• Conclusion

14. Edge Cases — Outlier Domains#

14.1 Code#

14.2 Math#

14.3 Emoji + Unicode#

14.4 Mixed-language (code-switching)#

14.5 URLs / emails#

14.6 Numbers#

14.7 Reserved tokens#

14.8 Whitespace pre-tokenization conflicts#

Egzersizler#

Egzersiz 1#

Egzersiz 2#

Egzersiz 3#

Egzersiz 4#

Egzersiz 5#

Egzersiz 6#

Egzersiz 7#

Egzersiz 8#

Egzersiz 9#

Egzersiz 10#

Sıradaki Ders: Capstone TurkTokenizer-tr#