HuggingFace Tokenizers Rust + Production Pipeline: Üretim-Kalite Tokenizer'ı Sıfırdan Eğitmek

Ders Haritası (16 Bölüm)#

1. HuggingFace ekosistemi — transformers vs tokenizers library ayrımı
2. Niye Rust — performans, safety, ekosistem entegrasyonu
3. Pipeline mimarisi — 6 katman ve veri akışı
4. Normalizers — NFC/NFD/NFKC/NFKD, Lowercase, StripAccents, BertNormalizer, Sequence
5. PreTokenizers — Whitespace, ByteLevel, Metaspace, Digits, Split, UnicodeScripts
6. Models — BPE, WordPiece, Unigram, WordLevel — algorithm parametreleri
7. PostProcessors — TemplateProcessing, BertProcessing, RobertaProcessing, ByteLevel
8. Decoders — round-trip lossless decoding
9. Trainers — BpeTrainer, WordPieceTrainer, UnigramTrainer config detayları
10. End-to-end Türkçe training — corpus → tokenizer.json (production-grade)
11. tokenizer.json format — JSON şeması, vocab merge_rules, byte-by-byte
12. Loading & saving — pretrained, conversion, custom
13. Rust internals — parallel processing, ahash, SIMD, mmap, lock-free design
14. Production deployment — caching, threading, FFI overhead, memory profile
15. Conversion — tiktoken → HF, SentencePiece → HF, custom formats
16. Benchmarks — fertility, throughput, memory, Trendyol-LLM karşılaştırma

1. HuggingFace Ekosistemi — Library Ayrımı#

1.1 `transformers` library (model)#

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

1.2 `tokenizers` library (tokenization)#

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

1.3 `tiktoken` library (OpenAI native)#

1.4 `sentencepiece` library (Google)#

1.5 Tercih kriteri#

1.6 İndirme & install#

pip install tokenizers          # Rust core, fast
pip install transformers        # Otomatik tokenizers'ı çeker
pip install sentencepiece       # T5, mT5 için
pip install tiktoken            # OpenAI için

1.7 Versioning#

• `tokenizers` 0.20+ (2024 ortası): Unigram trainer Rust'a taşındı, %50 hız artışı.
• `tokenizers` 0.21+ (2025): Multimodal token desteği eklendi.
• `transformers` her zaman uyumlu `tokenizers` versiyonu bağımlısı (dependency pin).

2. Niye Rust#

2.1 Performans nümerik gerçeği#

• Compiler optimization: Rust LLVM ile aggressive optimize
• Type safety: runtime type check yok
• Cache locality: contiguous memory, predictable layout
• SIMD: byte processing'de vectorization (avx2, neon)
• Lock-free hashmap: `ahash` + thread-local lookup

2.2 Memory safety#

• Buffer overflow risk = production crash
• Manual memory management hata-yatkın
• Rust ownership system → derleme zamanı garantiler

2.3 Ekosistem#

• pyo3 / maturin: Python binding generator, mükemmel UX
• napi-rs: Node.js binding (HF tokenizers WASM bridge için)
• JNI: Java binding (production servers için)
• C ABI: any language

2.4 Async ready#

2.5 Karşılaştırma: BPE encoding 10K cümle#

Pure Python (Karpathy minbpe):    24.3 sec
SentencePiece (C++):              1.4 sec
tiktoken (Rust):                  0.18 sec
tokenizers (Rust):                0.22 sec  (HF, biraz fazla overhead)

3. 6 Katmanlı Pipeline Mimarisi#

3.1 Veri akışı#

Raw text
    ↓ Normalizer
Normalized text (str)
    ↓ PreTokenizer
List of (str, (start, end))   # Pre-tokens with byte offsets
    ↓ Model (BPE/WordPiece/Unigram/WordLevel)
List of (token_id, (start, end))   # Final tokens
    ↓ PostProcessor
List of (token_id, (start, end))   # With special tokens added
    ↓ [Decoder for reverse direction]
str   # Decoded text

3.2 Her katmanın görevi#

Normalizer

PreTokenizer

Model

PostProcessor

Decoder

Trainer

3.3 Pipeline immutable mı?#

tokenizer.normalizer = NFD()
tokenizer.pre_tokenizer = Whitespace()
# ...

3.4 Her katman opsiyonel#

# Minimal tokenizer: sadece Model (WordLevel)
tokenizer = Tokenizer(WordLevel(unk_token="[UNK]"))
# Normalizer yok, PreTokenizer yok, PostProcessor yok

4. Normalizers — Unicode Hijyeni#

4.1 NFC / NFD / NFKC / NFKD#

1. U+00E9 (precomposed)
2. U+0065 + U+0301 (e + combining acute) — decomposed

• NFC: Composition. Tek karakter halinde (U+00E9). Production'da en yaygın.
• NFD: Decomposition. Birden çok karakter (U+0065 U+0301). StripAccents'ten önce.
• NFKC: Compatibility composition. "fi" (U+FB01) → "fi". Daha agresif.
• NFKD: Compatibility decomposition.

from tokenizers.normalizers import NFC, NFD, NFKC, NFKD
tokenizer.normalizer = NFC()

4.2 Lowercase#

from tokenizers.normalizers import Lowercase
tokenizer.normalizer = Lowercase()

4.3 StripAccents#

from tokenizers.normalizers import NFD, StripAccents, Sequence
tokenizer.normalizer = Sequence([NFD(), StripAccents()])

4.4 BertNormalizer#

from tokenizers.normalizers import BertNormalizer
tokenizer.normalizer = BertNormalizer(
    clean_text=True,         # control chars temizle
    handle_chinese_chars=True,  # Çinli karakterler arasına space
    strip_accents=None,      # None = lower case için strip, ya da False
    lowercase=True,
)

• `strip_accents=False` (Türkçe spesifik karakterleri koru)
• `lowercase=True` veya `False` (cased model istiyorsan False)
• BERT-base-Turkish-cased: `lowercase=False, strip_accents=False`
• BERT-base-Turkish-uncased: `lowercase=True, strip_accents=False`

4.5 Replace#

from tokenizers.normalizers import Replace
tokenizer.normalizer = Replace(pattern=r"\s+", content=" ")  # multiple spaces → single

4.6 Sequence (compose)#

from tokenizers.normalizers import Sequence, NFC, Replace, Lowercase
tokenizer.normalizer = Sequence([
    NFC(),
    Replace(r"\s+", " "),
    Lowercase(),
])

4.7 Strip, Prepend#

from tokenizers.normalizers import Strip, Prepend
Strip(left=True, right=True)  # whitespace trim
Prepend(prepend=" ")           # SentencePiece-style ▁ benzeri

4.8 Türkçe production önerisi#

from tokenizers.normalizers import Sequence, NFC, Replace
tokenizer.normalizer = Sequence([
    NFC(),                              # Unicode normalize
    Replace(r"[\u200B-\u200F]", ""),    # zero-width chars temizle
    Replace(r"\s+", " "),                # whitespace collapse
    Strip(),                             # trim
])

5. PreTokenizers — Boundary Çizenler#

5.1 Whitespace#

from tokenizers.pre_tokenizers import Whitespace
tokenizer.pre_tokenizer = Whitespace()
# "Merhaba, dünya!" → ["Merhaba", ",", "dünya", "!"]

5.2 WhitespaceSplit#

from tokenizers.pre_tokenizers import WhitespaceSplit
tokenizer.pre_tokenizer = WhitespaceSplit()
# "Merhaba, dünya!" → ["Merhaba,", "dünya!"]

5.3 Punctuation#

from tokenizers.pre_tokenizers import Punctuation
tokenizer.pre_tokenizer = Punctuation()
# "Merhaba,dünya!" → ["Merhaba", ",", "dünya", "!"]

5.4 ByteLevel — GPT-2/4 stili#

from tokenizers.pre_tokenizers import ByteLevel
tokenizer.pre_tokenizer = ByteLevel(
    add_prefix_space=True,    # " Merhaba" tek token (space prefix)
    use_regex=True,           # GPT-2 regex pattern
    trim_offsets=True,         # offset cleanup
)

5.5 Metaspace — SentencePiece stili#

from tokenizers.pre_tokenizers import Metaspace
tokenizer.pre_tokenizer = Metaspace(
    replacement="▁",          # U+2581 lower one eighth block
    prepend_scheme="first",   # ya da "always" / "never"
)

5.6 Digits#

from tokenizers.pre_tokenizers import Digits
tokenizer.pre_tokenizer = Digits(individual_digits=True)
# "2026 yılı" → ["2", "0", "2", "6", " yılı"]

5.7 Split — custom delimiter#

from tokenizers.pre_tokenizers import Split
tokenizer.pre_tokenizer = Split(
    pattern=r"[A-Z][a-z]+",
    behavior="isolated",  # ya da "removed" / "merged_with_previous" / "merged_with_next"
    invert=False,
)

5.8 CharDelimiterSplit#

CharDelimiterSplit(delimiter="|")

5.9 BertPreTokenizer#

from tokenizers.pre_tokenizers import BertPreTokenizer
tokenizer.pre_tokenizer = BertPreTokenizer()

5.10 UnicodeScripts#

from tokenizers.pre_tokenizers import UnicodeScripts
tokenizer.pre_tokenizer = UnicodeScripts()

5.11 Sequence (compose)#

from tokenizers.pre_tokenizers import Sequence, Whitespace, Punctuation, Digits
tokenizer.pre_tokenizer = Sequence([
    Whitespace(),
    Punctuation(),
    Digits(individual_digits=True),
])

5.12 Türkçe için önerim#

from tokenizers.pre_tokenizers import ByteLevel
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True, use_regex=True)

6. Models — Algoritma Çekirdeği#

6.1 BPE#

from tokenizers.models import BPE
model = BPE(
    vocab=None,           # dict: {token: id}
    merges=None,          # list: [("a", "b"), ...]
    unk_token="<unk>",
    byte_fallback=True,   # rare chars → bytes (Llama-3 stili)
    dropout=None,         # BPE-dropout (Provilkov 2020)
    fuse_unk=False,
    end_of_word_suffix=None,
    continuing_subword_prefix=None,
)

• byte_fallback: True → asla UNK, rare chars byte-level fallback.
• dropout: 0.0-1.0. Training'te %X merge'i skip (regularization).
• continuing_subword_prefix: "##" (WordPiece-style ama BPE'de) — pre-tokenize word continuation marker.
• end_of_word_suffix: "" (eski BPE word boundary marker).

6.2 WordPiece#

from tokenizers.models import WordPiece
model = WordPiece(
    vocab=None,
    unk_token="[UNK]",
    max_input_chars_per_word=100,    # uzun kelimeler için limit
    continuing_subword_prefix="##",
)

6.3 Unigram#

from tokenizers.models import Unigram
model = Unigram(
    vocab=None,           # list: [(token, log_prob), ...]
    unk_id=None,
    byte_fallback=False,
)

6.4 WordLevel — basit#

from tokenizers.models import WordLevel
model = WordLevel(
    vocab=None,           # dict
    unk_token="<unk>",
)

6.5 Algorithm seçim matrisi#

6.6 Türkçe için tercih#

7. PostProcessors — Special Token Enjeksiyonu#

7.1 TemplateProcessing#

from tokenizers.processors import TemplateProcessing
tokenizer.post_processor = TemplateProcessing(
    single="[CLS] $A [SEP]",
    pair="[CLS] $A [SEP] $B:1 [SEP]:1",
    special_tokens=[
        ("[CLS]", tokenizer.token_to_id("[CLS]")),
        ("[SEP]", tokenizer.token_to_id("[SEP]")),
    ],
)

• `A\`, \`B`: input sequence A ve B (pair tasks için)
• `:0`, `:1`: token_type_id (BERT segment ID)
• `[CLS]`, `[SEP]`: literal special tokenlar

7.2 BertProcessing — shortcut#

from tokenizers.processors import BertProcessing
tokenizer.post_processor = BertProcessing(
    sep=("[SEP]", tokenizer.token_to_id("[SEP]")),
    cls=("[CLS]", tokenizer.token_to_id("[CLS]")),
)

7.3 RobertaProcessing#

from tokenizers.processors import RobertaProcessing
tokenizer.post_processor = RobertaProcessing(
    sep=("</s>", tokenizer.token_to_id("</s>")),
    cls=("<s>", tokenizer.token_to_id("<s>")),
    add_prefix_space=True,
)

7.4 ByteLevel#

from tokenizers.processors import ByteLevel
tokenizer.post_processor = ByteLevel(trim_offsets=True)

7.5 Custom chat template — direkt PostProcessor değil#

8. Decoders — Round-Trip Lossless#

8.1 BPE Decoder#

from tokenizers.decoders import BPEDecoder
tokenizer.decoder = BPEDecoder()

8.2 WordPiece Decoder#

from tokenizers.decoders import WordPiece
tokenizer.decoder = WordPiece(prefix="##", cleanup=True)

8.3 ByteLevel Decoder#

from tokenizers.decoders import ByteLevel
tokenizer.decoder = ByteLevel()

8.4 Metaspace Decoder#

from tokenizers.decoders import Metaspace
tokenizer.decoder = Metaspace(replacement="▁")

8.5 Lossless requirement#

• ByteLevel BPE: ✅ tam lossless
• WordPiece: ⚠️ "##" prefix bilgi taşır ama whitespace handling değişebilir
• Metaspace: ✅ ▁ → space tam reversible
• BPE classic: ⚠️ word boundary marker depending on training

8.6 Strip, Replace, Sequence#

from tokenizers.decoders import Strip, Replace, Sequence
Sequence([ByteLevel(), Replace("▁", " "), Strip(content=" ", left=1)])

9. Trainers — Vocab Öğrenme#

9.1 BpeTrainer#

from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(
    vocab_size=32000,
    min_frequency=2,             # min pair count to consider
    special_tokens=["<pad>", "<unk>", "<s>", "</s>", "<|im_start|>", "<|im_end|>"],
    initial_alphabet=ByteLevel.alphabet(),  # tüm 256 byte
    continuing_subword_prefix=None,
    end_of_word_suffix=None,
    show_progress=True,
    max_token_length=None,
)

• vocab_size: target. Türkçe-only 32K-50K, multilingual 100K+.
• min_frequency: 2 default. Küçük corpus için 1.
• special_tokens: vocab başına eklenir, ID 0,1,2,... reserved.
• initial_alphabet: base chars/bytes. ByteLevel.alphabet() → 256 byte garanti.

9.2 WordPieceTrainer#

from tokenizers.trainers import WordPieceTrainer
trainer = WordPieceTrainer(
    vocab_size=32000,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
    continuing_subword_prefix="##",
    end_of_word_suffix=None,
    show_progress=True,
)

9.3 UnigramTrainer#

from tokenizers.trainers import UnigramTrainer
trainer = UnigramTrainer(
    vocab_size=32000,
    special_tokens=["<pad>", "<unk>", "<s>", "</s>"],
    initial_alphabet=[],
    shrinking_factor=0.75,           # round başına %25 küçült
    unk_token="<unk>",
    max_piece_length=16,
    n_sub_iterations=2,
    seed_sentencepiece_size=1_000_000,
    show_progress=True,
)

9.4 WordLevelTrainer#

from tokenizers.trainers import WordLevelTrainer
trainer = WordLevelTrainer(
    vocab_size=10000,
    min_frequency=5,
    special_tokens=["<unk>"],
)

9.5 Trainer çalıştırma#

files = ["corpus/wiki-tr.txt", "corpus/news-tr.txt"]
tokenizer.train(files, trainer)

def batch_iterator():
    for batch in dataset.iter(batch_size=1000):
        yield batch["text"]

tokenizer.train_from_iterator(batch_iterator(), trainer=trainer, length=len(dataset))

9.6 Threading#

import os
os.environ["RAYON_NUM_THREADS"] = "16"  # explicit thread sayısı

10. End-to-End Türkçe Tokenizer Training (Production-Grade)#

Adım adım kodun:#

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.normalizers import Sequence, NFC, Replace, Strip
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.processors import ByteLevel as ByteLevelProcessor
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.trainers import BpeTrainer

# 1. Tokenizer init
tokenizer = Tokenizer(BPE(unk_token=None, byte_fallback=True))

# 2. Normalizer
tokenizer.normalizer = Sequence([
    NFC(),
    Replace(r"[\u200B-\u200F\uFEFF]", ""),   # zero-width chars
    Replace(r"\s+", " "),                       # whitespace collapse
    Strip(),
])

# 3. Pre-tokenizer
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True, use_regex=True)

# 4. Post-processor
tokenizer.post_processor = ByteLevelProcessor(trim_offsets=True)

# 5. Decoder
tokenizer.decoder = ByteLevelDecoder()

# 6. Trainer config
trainer = BpeTrainer(
    vocab_size=32000,
    min_frequency=2,
    special_tokens=[
        "<|endoftext|>",
        "<|pad|>",
        "<|im_start|>",
        "<|im_end|>",
        "<|user|>",
        "<|assistant|>",
        "<|system|>",
        "<|tool|>",
    ],
    initial_alphabet=ByteLevel.alphabet(),
    show_progress=True,
    max_token_length=24,    # max sub-word length
)

# 7. Train
files = [
    "corpus/wiki-tr.txt",
    "corpus/news-tr.txt",
    "corpus/oscar-tr.txt",
    "corpus/literature-tr.txt",
]
tokenizer.train(files, trainer)

# 8. Save
tokenizer.save("turk-bpe-32k.json")

# 9. Test
result = tokenizer.encode("İstanbul Boğazı'nda balıkçı tekneleri sallanıyor.")
print("Tokens:", result.tokens)
print("IDs:", result.ids)
print("Count:", len(result.ids))

Training time benchmark#

• Step 1-5 (setup): <1 sec
• Step 7 (BPE training): ~25 minutes
• Total: ~25 minutes

Memory profile#

• Corpus mmap: ~10 GB (virtual, disk-backed)
• BPE training peak: ~12 GB RAM
• Output tokenizer.json: ~2 MB

Production önerileri#

1. Corpus diversity: Wikipedia + OSCAR + news + literature + code mix
2. Cleaning: HTML strip, deduplicate, language detect
3. Min frequency: 2 ideal, 1 only for very small corpus
4. Special tokens: chat format'ın hepsi vocab'da olsun
5. max_token_length: 16-24 sweet spot
6. initial_alphabet: byte-level garanti için ByteLevel.alphabet()

11. tokenizer.json Format Anatomisi#

11.1 Üst-seviye yapı#

{
  "version": "1.0",
  "truncation": null,
  "padding": null,
  "added_tokens": [...],
  "normalizer": {...},
  "pre_tokenizer": {...},
  "post_processor": {...},
  "decoder": {...},
  "model": {...}
}

11.2 added_tokens#

"added_tokens": [
  {
    "id": 0,
    "content": "<|endoftext|>",
    "single_word": false,
    "lstrip": false,
    "rstrip": false,
    "normalized": false,
    "special": true
  },
  {
    "id": 1,
    "content": "<|im_start|>",
    ...
  }
]

11.3 normalizer#

"normalizer": {
  "type": "Sequence",
  "normalizers": [
    {"type": "NFC"},
    {"type": "Replace", "pattern": {"Regex": "\\s+"}, "content": " "}
  ]
}

11.4 pre_tokenizer#

"pre_tokenizer": {
  "type": "ByteLevel",
  "add_prefix_space": true,
  "trim_offsets": true,
  "use_regex": true
}

11.5 model (BPE)#

"model": {
  "type": "BPE",
  "dropout": null,
  "unk_token": null,
  "continuing_subword_prefix": null,
  "end_of_word_suffix": null,
  "fuse_unk": false,
  "byte_fallback": true,
  "vocab": {
    "!": 0,
    "\"": 1,
    ...
    "Ġmerhaba": 4523,
    "Ġdünya": 6711,
    ...
  },
  "merges": [
    "Ġ d",
    "e r",
    "i n",
    ...
  ]
}

11.6 vocab byte encoding#

11.7 merges sırası kritik#

11.8 Dosya boyutu beklenen#

• 32K vocab BPE: ~2-3 MB
• 100K vocab: ~6-8 MB
• 200K vocab (o200k): ~12 MB

11.9 Sürüm uyumluluğu#

12. Loading & Saving — Format Galaxies#

12.1 HF native#

# Save
tokenizer.save("turk-bpe-32k.json")

# Load
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("turk-bpe-32k.json")

12.2 HF Hub'dan pretrained#

tokenizer = Tokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

12.3 transformers AutoTokenizer ile uyumlu#

from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast(tokenizer_file="turk-bpe-32k.json")
tokenizer.save_pretrained("./my-tokenizer")
# Çıktı: my-tokenizer/tokenizer.json + tokenizer_config.json + special_tokens_map.json

12.4 Legacy BERT format (vocab.txt)#

from tokenizers import BertWordPieceTokenizer
tokenizer = BertWordPieceTokenizer("vocab.txt", lowercase=False)

12.5 Legacy SentencePiece (.model)#

# transformers auto-converts SP .model to fast tokenizer
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("google/mt5-base")
fast = tokenizer.train_new_from_iterator(...)

12.6 tiktoken → HF conversion#

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

# 1. Vocab extract
vocab = {}
for i in range(enc.n_vocab):
    try:
        token = enc.decode_single_token_bytes(i).decode("utf-8", errors="replace")
        vocab[token] = i
    except:
        pass

# 2. HF BPE oluştur (merges yok!)
from tokenizers import Tokenizer
from tokenizers.models import BPE
tokenizer = Tokenizer(BPE(vocab=vocab, merges=[], byte_fallback=True))
# Note: merges yok → bu sadece encode yapamayan dummy. Tam conversion için merge inferring gerekir.

13. Rust Internals — Performansın Kaynakları#

13.1 Parallel processing (rayon)#

// Rust internals (simplified)
use rayon::prelude::*;

pub fn encode_batch(&self, inputs: Vec<&str>) -> Vec<Encoding> {
    inputs.par_iter()
        .map(|input| self.encode(input))
        .collect()
}

results = tokenizer.encode_batch(["cümle 1", "cümle 2", "cümle 3"])

13.2 ahash — fast hashmap#

• 2-3x daha hızlı lookup
• Vocab lookup'da kritik (her char için)
• Production-tested, no DoS issue (tokenizer input trusted)

13.3 SIMD vectorization#

• AVX2 (x86_64): 32 byte at once
• NEON (ARM, Apple Silicon): 16 byte at once
• UTF-8 validation, byte-to-unicode mapping, BPE merge application — hepsi SIMD-friendly

13.4 mmap (memory-mapped I/O)#

• OS page cache leverage
• Disk I/O bandwidth saturate (NVMe için ~3 GB/sec)
• RAM kullanımı virtual (OS yönetir)

13.5 Lock-free design#

13.6 Zero-copy strings#

13.7 Profile detayı (encode 1M cümle)#

Normalizer:       12% time
PreTokenizer:     8%
BPE lookup:       65% time (vocab dictionary lookup)
PostProcessor:    3%
Overhead:         12% (FFI, allocation, etc.)

14. Production Deployment — Pratik Konular#

14.1 Threading#

import os
os.environ["RAYON_NUM_THREADS"] = "4"   # Kubernetes pod 4 CPU

14.2 Caching#

• Redis cache (hash(text) → tokens)
• LRU local cache (Python `functools.lru_cache`)
• Database column (her satır için precomputed tokens)

14.3 FFI overhead#

# YAVAŞ — 1M call x 5 μs = 5 sec overhead
for sentence in corpus:
    tokens = tokenizer.encode(sentence)

# HIZLI — 1 call, batch processing
tokens_batch = tokenizer.encode_batch(corpus)

14.4 Memory profile#

14.5 Multi-process worker pattern#

from multiprocessing import Pool

def worker(text):
    return tokenizer.encode(text)

with Pool(8) as p:
    results = p.map(worker, corpus)

14.6 vLLM / TGI integration#

14.7 Streaming (real-time)#

for token_id in model_output_stream:
    text = tokenizer.decode([token_id])
    print(text, end="", flush=True)

15. Conversion — Diğer Formatlardan HF'ye#

15.1 tiktoken → HF#

15.2 SentencePiece → HF#

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
# Internal: SentencePiece .model → HF tokenizer.json
tokenizer.save_pretrained("./mt5-tokenizer")

15.3 OpenAI'dan kendi format'a#

15.4 BERT vocab.txt → HF#

from tokenizers import BertWordPieceTokenizer
tokenizer = BertWordPieceTokenizer("vocab.txt", lowercase=False)
tokenizer.save("bert-tr.json")

15.5 GPT-2 encoder.json + vocab.bpe → HF#

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors
import json

with open("encoder.json") as f:
    vocab = json.load(f)

with open("vocab.bpe") as f:
    merges = [tuple(line.split()) for line in f.readlines()[1:] if line.strip()]

tokenizer = Tokenizer(models.BPE(vocab=vocab, merges=merges))
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=False)
tokenizer.save("gpt2-hf.json")

16. Benchmarks — Türkçe Karşılaştırma#

16.1 Fertility (token/word) — 1 sayfa Türkçe metin (300 kelime)#

16.2 Encoding throughput (single thread)#

16.3 Batch encoding (16 thread)#

16.4 Memory#

16.5 Training time (10 GB Türkçe corpus, 16 core)#

• BPE: ~25 min
• WordPiece: ~30 min
• Unigram: ~90 min (slower algorithm)

16.6 İndirme & startup time#

• Tokenizer.from_file(): ~50-200 ms (vocab parsing)
• from_pretrained(): ilk kez ~3 sec (network + parse), cached ~100 ms

Egzersizler#

Egzersiz 1#

Egzersiz 2#

Egzersiz 3#

Egzersiz 4#

Egzersiz 5#

Egzersiz 6#

Egzersiz 7#

Egzersiz 8#

Egzersiz 9#

Egzersiz 10#

Sıradaki Ders: Tokenizer Evaluation#