Capstone TurkTokenizer-tr: Train, Evaluate, and Publish a Production-Grade Turkish Tokenizer to HuggingFace Hub

Capstone Akışı (12 Aşama)#

1. Hedef tanımı — niye TurkTokenizer-tr, kim için, hangi metric'le başarı
2. Corpus curation — Wikipedia + OSCAR + news + literature + code mix
3. Cleaning pipeline — HTML strip, dedup, language detect, quality filter
4. Training config — 32K BPE, byte_fallback, special tokens (chat-ready)
5. End-to-end training script — production-grade, 25 min on 16-core
6. Evaluation (6.9 framework) — fertility, BPC, OOV, downstream proxy
7. Comparison — Llama-3, GPT-4o, Trendyol-LLM, mBERT
8. Model card yazımı — Hugging Face standartına uygun
9. License seçimi — MIT vs Apache 2.0 vs CC-BY-SA, hangisi ne zaman
10. HuggingFace Hub workflow — repo oluşturma, upload, versioning
11. Production integration — transformers, vLLM, kendi pipeline'ın
12. Maintenance roadmap — versioning, community feedback, retraining schedule

1. Hedef Tanımı#

1.1 TurkTokenizer-tr nedir#

• İsim: TurkTokenizer-tr (community-friendly, descriptive)
• Algoritma: BPE (byte-level)
• Vocab boyutu: 32,000
• byte_fallback: True
• Special tokens: chat-ready (ChatML compat)
• Hedef dil: Türkçe (TR-only fine-tune'lar için)
• License: Apache 2.0 (commercial-friendly)

1.2 Kim için#

1. Türkçe LLM fine-tune yapanlar (Llama-3'ün vocab'ını değiştirmek istemeyenler için base model swap alternatif)
2. Türkçe pre-training projeleri
3. Türkçe NER/QA/sentiment fine-tune (BERT-style)
4. Türkçe maliyet optimizasyonu peşinde olanlar
5. Akademik karşılaştırma için

1.3 Başarı kriterleri#

• Fertility ≤ 1.25 (Wikipedia TR test set)
• BPC ≤ 0.85 (proxy LM ile)
• OOV rate ≤ %25
• Cross-domain fertility ≤ 1.5 (legal, medical hariç)
• HuggingFace Hub'da indexed + downloadable
• Model card complete, reproducible

1.4 Zaman çizelgesi#

• Corpus toplama: 4 saat
• Cleaning: 2 saat
• Training: 30 dakika
• Evaluation: 1 saat
• Model card + license: 2 saat
• Hub publish: 1 saat
• Total: 1 günlük capstone

1.5 Önceden var olan benzer projeler#

• Trendyol-LLM tokenizer (closed, comparable)
• BERT-base-Turkish-cased (WordPiece, BERT-style)
• mukayese (Türkçe NLP toolkit, fragmente)

2. Corpus Curation#

2.1 Kaynak seçimi (10 GB hedef)#

2.2 Download script#

from datasets import load_dataset

# Wikipedia
wiki = load_dataset("wikipedia", "20240401.tr", split="train")
wiki.to_parquet("corpus/wiki-tr.parquet")

# OSCAR
oscar = load_dataset("oscar-corpus/OSCAR-2301", "tr", split="train", streaming=True)
with open("corpus/oscar-tr.txt", "w", encoding="utf-8") as f:
    for i, item in enumerate(oscar):
        if i >= 5_000_000:  # ~5 GB
            break
        f.write(item["text"] + "\n")

2.3 License compliance#

• Wikipedia (CC-BY-SA): attribution gerekli, derived work share-alike
• OSCAR (CC0): no restriction, ideal
• Code (mixed): filter permissive licenses (MIT, Apache, BSD) — GPL exclude
• News (Academic): research/educational only — commercial use için ayrıca

2.4 Domain balance#

• Wikipedia: %40
• OSCAR (general web): %35
• News: %10
• Literature: %5
• Code: %5
• Other: %5

2.5 Türkçe-spesifik considerations#

• Ünlü uyumu coverage: tüm sufiks variantları temsil edilsin
• Mixed Latin script: İngilizce loan-words, technical terms
• Modern + Ottoman: optional Ottoman corpus for historical text support
• Dialects: standart İstanbul Türkçesi tercih, dialektler under-representation OK
• Apostrof: 'da, 'nın gibi suffixleri ortaya çıkaran corpus seç

3. Cleaning Pipeline#

3.1 Steps#

import re
import html
from langdetect import detect

def clean_pipeline(text):
    # 1. HTML strip
    text = re.sub(r"<[^>]+>", "", text)
    text = html.unescape(text)
    
    # 2. Whitespace normalize
    text = re.sub(r"\s+", " ", text)
    text = text.strip()
    
    # 3. Min length filter
    if len(text) < 50:
        return None
    
    # 4. Language detect
    try:
        if detect(text) != "tr":
            return None
    except:
        return None
    
    # 5. Quality filter (alphanumeric ratio)
    alnum = sum(1 for c in text if c.isalnum())
    if alnum / len(text) < 0.5:
        return None
    
    # 6. PII filter (email, phone — basit)
    text = re.sub(r"\S+@\S+", "<EMAIL>", text)
    text = re.sub(r"\+?\d[\d\s\-]{8,}\d", "<PHONE>", text)
    
    return text

3.2 Deduplication#

import hashlib
seen_hashes = set()

def dedupe(text):
    h = hashlib.md5(text.encode("utf-8")).hexdigest()
    if h in seen_hashes:
        return None
    seen_hashes.add(h)
    return text

3.3 Pipeline orchestration#

def process_corpus(input_files, output_file):
    with open(output_file, "w", encoding="utf-8") as out:
        for fname in input_files:
            with open(fname, encoding="utf-8") as f:
                for line in f:
                    cleaned = clean_pipeline(line)
                    if cleaned is None:
                        continue
                    deduped = dedupe(cleaned)
                    if deduped is None:
                        continue
                    out.write(deduped + "\n")

process_corpus(
    ["corpus/wiki-tr.txt", "corpus/oscar-tr.txt", "corpus/news-tr.txt"],
    "corpus/clean-tr.txt"
)

3.4 Quality metrics#

import collections

def corpus_stats(file):
    line_lengths = []
    word_counts = collections.Counter()
    with open(file, encoding="utf-8") as f:
        for line in f:
            line_lengths.append(len(line))
            for word in line.split():
                word_counts[word] += 1
    
    print(f"Lines: {len(line_lengths):,}")
    print(f"Avg line length: {sum(line_lengths)/len(line_lengths):.1f}")
    print(f"Unique words: {len(word_counts):,}")
    print(f"Total words: {sum(word_counts.values()):,}")
    
    return word_counts

3.5 Expected output#

3.6 Modern alternative: HF datasets dedupe#

from datasets import load_dataset
ds = load_dataset("text", data_files=["corpus/clean-tr.txt"])
ds = ds.unique("text")  # exact dedup
ds.to_csv("corpus/final-tr.csv")

4. Training Config#

4.1 Hyperparameters#

VOCAB_SIZE = 32000
MIN_FREQUENCY = 2
MAX_TOKEN_LENGTH = 24

SPECIAL_TOKENS = [
    "<|endoftext|>",       # GPT-style
    "<|pad|>",
    "<|im_start|>",        # ChatML
    "<|im_end|>",
    "<|user|>",
    "<|assistant|>",
    "<|system|>",
    "<|tool|>",
    "<|python_tag|>",      # tool use
    "<|eot_id|>",          # Llama-3 compat
    "<EMAIL>",             # cleaning placeholder
    "<PHONE>",
    "<URL>",
    "<NUMBER>",
]

4.2 Why these choices#

• 32K: Türkçe-only optimum. 50K marjinal kazanç (-%5 fertility), 2x embedding maliyeti.
• byte_fallback=True: Llama-3 pattern. UNK eliminate.
• special_tokens chat-ready: production fine-tune'larda direkt kullanım.
• ChatML + Llama-3 special tokens her ikisi: cross-compat.

4.3 Normalizer#

from tokenizers.normalizers import Sequence, NFC, Replace, Strip

normalizer = Sequence([
    NFC(),
    Replace(r"[\u200B-\u200F\uFEFF\u2028\u2029]", ""),   # zero-width + line sep
    Replace(r"\s+", " "),
    Strip(),
])

4.4 PreTokenizer#

from tokenizers.pre_tokenizers import ByteLevel
pre_tokenizer = ByteLevel(add_prefix_space=True, use_regex=True)

4.5 Post-processor#

from tokenizers.processors import ByteLevel as ByteLevelProcessor
post_processor = ByteLevelProcessor(trim_offsets=True)

4.6 Decoder#

from tokenizers.decoders import ByteLevel as ByteLevelDecoder
decoder = ByteLevelDecoder()

4.7 Trainer#

from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(
    vocab_size=VOCAB_SIZE,
    min_frequency=MIN_FREQUENCY,
    special_tokens=SPECIAL_TOKENS,
    initial_alphabet=ByteLevel.alphabet(),
    show_progress=True,
    max_token_length=MAX_TOKEN_LENGTH,
)

4.8 Chat template#

CHAT_TEMPLATE = '''
{% for message in messages -%}
{%- if message.role == "system" -%}
<|im_start|>system
{{ message.content }}<|im_end|>
{%- elif message.role == "user" -%}
<|im_start|>user
{{ message.content }}<|im_end|>
{%- elif message.role == "assistant" -%}
<|im_start|>assistant
{{ message.content }}<|im_end|>
{%- endif -%}
{% endfor -%}
{%- if add_generation_prompt -%}
<|im_start|>assistant
{%- endif -%}
'''

5. End-to-End Training Script (Production-Grade)#

#!/usr/bin/env python3
# train_turktokenizer.py

import os
import time
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.normalizers import Sequence, NFC, Replace, Strip
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.processors import ByteLevel as ByteLevelProcessor
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.trainers import BpeTrainer

# Config
VOCAB_SIZE = 32000
MIN_FREQUENCY = 2
MAX_TOKEN_LENGTH = 24
CORPUS_FILES = [
    "corpus/wiki-tr.txt",
    "corpus/oscar-tr-clean.txt",
    "corpus/news-tr.txt",
    "corpus/literature-tr.txt",
    "corpus/code-tr.txt",
]
OUTPUT_PATH = "turktokenizer-tr-32k.json"

SPECIAL_TOKENS = [
    "<|endoftext|>",
    "<|pad|>",
    "<|im_start|>",
    "<|im_end|>",
    "<|user|>",
    "<|assistant|>",
    "<|system|>",
    "<|tool|>",
    "<|python_tag|>",
    "<|eot_id|>",
    "<EMAIL>",
    "<PHONE>",
    "<URL>",
    "<NUMBER>",
]


def main():
    print("🚀 TurkTokenizer-tr eğitimi başlıyor...")
    start = time.time()
    
    # 1. Tokenizer init
    tokenizer = Tokenizer(BPE(unk_token=None, byte_fallback=True))
    
    # 2. Normalizer
    tokenizer.normalizer = Sequence([
        NFC(),
        Replace(r"[\u200B-\u200F\uFEFF\u2028\u2029]", ""),
        Replace(r"\s+", " "),
        Strip(),
    ])
    
    # 3. Pre-tokenizer
    tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True, use_regex=True)
    
    # 4. Post-processor
    tokenizer.post_processor = ByteLevelProcessor(trim_offsets=True)
    
    # 5. Decoder
    tokenizer.decoder = ByteLevelDecoder()
    
    # 6. Trainer
    trainer = BpeTrainer(
        vocab_size=VOCAB_SIZE,
        min_frequency=MIN_FREQUENCY,
        special_tokens=SPECIAL_TOKENS,
        initial_alphabet=ByteLevel.alphabet(),
        show_progress=True,
        max_token_length=MAX_TOKEN_LENGTH,
    )
    
    # 7. Train
    tokenizer.train(CORPUS_FILES, trainer)
    
    # 8. Save
    tokenizer.save(OUTPUT_PATH)
    print(f"✅ Saved to {OUTPUT_PATH}")
    
    # 9. Smoke test
    test_text = "İstanbul Boğazı'nda balıkçı tekneleri sallanıyor."
    result = tokenizer.encode(test_text)
    print(f"Test text: {test_text}")
    print(f"Tokens ({len(result.ids)}): {result.tokens}")
    
    elapsed = time.time() - start
    print(f"⏱️  Training time: {elapsed:.0f} seconds")


if __name__ == "__main__":
    main()

5.1 Çalıştırma#

export RAYON_NUM_THREADS=16
python train_turktokenizer.py

5.2 Beklenen output#

🚀 TurkTokenizer-tr eğitimi başlıyor...
[1500/1500] 100% complete in 25:33
✅ Saved to turktokenizer-tr-32k.json
Test text: İstanbul Boğazı'nda balıkçı tekneleri sallanıyor.
Tokens (8): ['Ġİstanbul', 'ĠBoğazı', "'", 'nda', 'Ġbalıkçı', 'Ġtekneleri', 'Ġsallanıyor', '.']
⏱️  Training time: 1533 seconds

5.3 transformers compat#

from transformers import PreTrainedTokenizerFast

fast = PreTrainedTokenizerFast(tokenizer_file="turktokenizer-tr-32k.json")
fast.chat_template = CHAT_TEMPLATE  # önceki bölümden
fast.save_pretrained("./turktokenizer-tr-32k-hf")
# Çıktı: tokenizer.json + tokenizer_config.json + special_tokens_map.json

6. Evaluation — 6.9 Framework Uygulaması#

6.1 Test suites#

TEST_CORPORA = {
    "wiki_test": load_text("test/wiki-tr-test.txt"),       # 1M token
    "news_test": load_text("test/news-tr-test.txt"),       # 500K
    "legal_test": load_text("test/legal-tr-test.txt"),     # 200K
    "medical_test": load_text("test/medical-tr-test.txt"), # 100K
    "flores_tr": load_flores("tr"),                         # paralel cümleler
    "flores_en": load_flores("en"),                         # cross-lingual
}

6.2 Comparison tokenizers#

TOKENIZERS = {
    "turktokenizer-tr": Tokenizer.from_file("turktokenizer-tr-32k.json"),
    "llama-3": AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B"),
    "gpt-4o": tiktoken.get_encoding("o200k_base"),  # wrapper
    "bert-tr": AutoTokenizer.from_pretrained("dbmdz/bert-base-turkish-cased"),
}

6.3 Run#

results = {}
for tname, tok in TOKENIZERS.items():
    for cname, corpus in TEST_CORPORA.items():
        key = f"{tname}_{cname}"
        results[key] = evaluate_tokenizer(tok, corpus)

import pandas as pd
df = pd.DataFrame(results).T
df.to_csv("eval/results.csv")
print(df)

6.4 Expected metrics#

6.5 Cost analysis#

• Llama-3: 500 × 1M = 500M token (baseline)
• TurkTokenizer-tr: 500 × 1M × (1.22/1.47) = 415M token (-%17)
• Yıllık tasarruf (GPT-4o equivalent $2.5/1M rate):$213/ay = $2,550/yıl

6.6 Downstream task proxy (no full LM training)#

• Vocab utilization on Turkish corpus
• Subword coverage of common Turkish suffixes (-lar, -ler, -nın, -nin, -da, -de etc.)
• Boundary preservation for named entities (city names, person names)

turk_suffixes = ["lar", "ler", "nın", "nin", "nun", "nün", "da", "de", "dan", "den", "a", "e", "ı", "i", "u", "ü"]
coverage = sum(1 for s in turk_suffixes if s in tokenizer.get_vocab())
print(f"Turkish suffix coverage: {coverage}/{len(turk_suffixes)}")
# 16/16 ideal

7. Comparison — Detaylı Benchmark#

7.1 Fertility heatmap (görsel)#

7.2 Cost projection table#

7.3 Trendyol-LLM comparison#

• Trendyol-LLM: ~1.20 fertility Wikipedia
• TurkTokenizer-tr: ~1.22 fertility Wikipedia
• Comparable! TurkTokenizer-tr açık-kaynak alternative.

7.4 Qualitative analysis#

• Llama-3: ['Istanbul', 'Boğaz', 'ı', "'", 'nda', 'balık', 'çı', 'tekne', 'leri', 'sall', 'anıyor', '.'] (12 token)
• TurkTokenizer-tr: ['İstanbul', 'Boğazı', "'", 'nda', 'balıkçı', 'tekneleri', 'sallanıyor', '.'] (8 token)

7.5 Limitations#

• Code domain'de fertility yüksek (code-specific tokenizer gerek)
• Math/LaTeX coverage zayıf
• Multilingual değil (sadece TR)
• Dialects under-represented

8. Model Card — HuggingFace Hub Standardı#

8.1 Template (Markdown)#

---
language:
  - tr
license: apache-2.0
tags:
  - tokenizer
  - turkish
  - bpe
  - byte-level
  - llm
---

# TurkTokenizer-tr (32K BPE)

## Model Description
TurkTokenizer-tr is a 32,000-vocab byte-level BPE tokenizer trained specifically for Turkish language modeling. It achieves fertility of 1.22 tokens/word on Turkish Wikipedia, ~17% better than Llama-3 default tokenizer.

## Intended Use
- Turkish LLM fine-tuning
- Turkish BERT-style model training
- Cost optimization for Turkish LLM inference

## Training Data
- Turkish Wikipedia (4GB)
- OSCAR Turkish 2023 subset (5GB)
- Turkish news corpus (1GB)
- Turkish literature + code comments (1GB)
- Total: ~11GB clean Turkish text

## Training Procedure
- Algorithm: BPE (byte-level)
- Vocab size: 32,000
- byte_fallback: True
- Max token length: 24
- Min frequency: 2
- Training time: ~26 minutes on 16-core EPYC

## Evaluation
| Metric | Value |
|---|---|
| Fertility (Wiki TR) | 1.22 |
| Fertility (News TR) | 1.20 |
| BPC (proxy LM) | 0.83 |
| OOV rate | 24% |
| Cross-lingual (TR/EN) | 1.40 |

Detayed comparison: [eval/benchmarks.csv](./eval/benchmarks.csv)

## Special Tokens
- Chat: <|im_start|>, <|im_end|>, <|user|>, <|assistant|>, <|system|>
- Llama-3 compat: <|eot_id|>, <|python_tag|>
- Cleaning: <EMAIL>, <PHONE>, <URL>, <NUMBER>

## Limitations
- Code domain fertility higher (~2.8)
- Math/LaTeX coverage limited
- Monolingual (Turkish only)
- Trained on ISTANBUL Turkish standard

## License
Apache 2.0. Commercial use allowed. Training data CC-BY-SA compatible.

## Citation
```bibtex
@software{turktokenizer_tr_2026,
  author = {Şükrü Yusuf KAYA},
  title = {TurkTokenizer-tr: A 32K BPE Tokenizer for Turkish},
  year = {2026},
  url = {https://huggingface.co/sukruyusufkaya/turktokenizer-tr-32k}
}

Acknowledgments#

### 8.2 Required sections (HF guidelines)
- Model Description
- Intended Use
- Limitations
- Training Data
- Training Procedure
- Evaluation
- License
- Citation

### 8.3 Quality criteria
- Reproducible (link to training script)
- Quantitative metrics (not vague claims)
- Honest limitations
- Clear license + attribution

9. License Seçimi#

9.1 Common open-source licenses#

9.2 Choice for TurkTokenizer-tr#

• Permissive: commercial use OK
• Patent grant clause: çakışan patent riski yok
• Attribution gerekli ama virus-free (derived works restricted değil)
• Modern open-source standard
• HuggingFace ekosisteminde en yaygın

9.3 GPL niye değil#

9.4 Training data license uyumu#

• CC-BY-SA derived work share-alike
• AMA tokenizer model artifact (vocab + merges) derived work midir? Hukuken belirsiz alan.
• Bizim yorumumuz: tokenizer artifact tek başına creative değil — bir 'extraction' (kelime frekansları + bölünmeler).
• Apache 2.0 ile yayın defensible — Wikipedia community legal opinion benzer projelere izin veriyor.

9.5 Attribution metni (README'ye eklenir)#

10. HuggingFace Hub Workflow#

10.1 Account + token#

pip install huggingface_hub
huggingface-cli login
# Token: hf_xxxxx (HF settings'den alınır)

10.2 Repo oluşturma#

from huggingface_hub import HfApi
api = HfApi()
api.create_repo(
    repo_id="sukruyusufkaya/turktokenizer-tr-32k",
    repo_type="model",
    private=False,  # public release
)

10.3 Upload#

from huggingface_hub import upload_folder
upload_folder(
    folder_path="./turktokenizer-tr-32k-hf",
    repo_id="sukruyusufkaya/turktokenizer-tr-32k",
    commit_message="Initial release v1.0.0",
)

• tokenizer.json (~2 MB)
• tokenizer_config.json
• special_tokens_map.json
• README.md (model card)
• LICENSE (Apache 2.0 full text)
• eval/ (benchmark results)
• train_turktokenizer.py (training script)

10.4 Versioning#

api.create_tag(
    repo_id="sukruyusufkaya/turktokenizer-tr-32k",
    tag="v1.0.0",
    revision="main",
)

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
    "sukruyusufkaya/turktokenizer-tr-32k",
    revision="v1.0.0",  # specific version
)

10.5 Spaces — interactive demo (opsiyonel)#

import gradio as gr
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("sukruyusufkaya/turktokenizer-tr-32k")

def tokenize(text):
    ids = tok.encode(text)
    tokens = [tok.decode([i]) for i in ids]
    return ids, tokens, len(ids)

iface = gr.Interface(
    fn=tokenize,
    inputs="text",
    outputs=["text", "text", "number"],
    title="TurkTokenizer-tr Playground",
)
iface.launch()

10.6 Discoverability#

• HF tags: turkish, tokenizer, bpe, llm
• README iyi yazılmış → search rank
• Blog post (sukruyusufkaya.com) HF Hub'a link
• Twitter/LinkedIn announce
• Community: r/LocalLLaMA, HF forums, Türkiye AI Slack/Discord

11. Production Integration#

11.1 transformers ile yükleme#

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("sukruyusufkaya/turktokenizer-tr-32k")
print(tok.tokenize("Merhaba dünya"))

11.2 vLLM ile inference (custom model + tokenizer)#

from vllm import LLM
llm = LLM(
    model="sukruyusufkaya/turk-llama-3-8b-instruct",
    tokenizer="sukruyusufkaya/turktokenizer-tr-32k",  # opsiyonel — model içinde default
)
output = llm.generate(["Türkiye'nin başkenti nedir?"])

11.3 Llama-3 fine-tune ile uyum#

1. Pre-training from scratch: tokenizer + model birlikte. Pahalı.
2. Continued pre-training: Llama-3 base + Türkçe data + Llama-3 default tokenizer. Daha pratik.

11.4 Inference cost monitoring#

import time
start = time.time()
tokens = tok.encode("long text here...")
latency = time.time() - start
print(f"Latency: {latency*1000:.2f}ms, tokens: {len(tokens)}")

11.5 Caching#

from functools import lru_cache

@lru_cache(maxsize=10000)
def cached_encode(text):
    return tuple(tok.encode(text))

11.6 sukruyusufkaya.com integration#

• Site'de 'tools' bölümüne TurkTokenizer-tr playground ekle
• Real-time tokenize web component (HF widget embed)
• Türkçe maliyet hesaplayıcı (cl100k vs TurkTokenizer comparison)

12. Maintenance Roadmap#

12.1 Versioning strategy#

• MAJOR: breaking changes (vocab değişir → embedding incompat)
• MINOR: backwards-compat additions (yeni special token)
• PATCH: bug fixes (normalization regex fix)

• v1.0.1: cleanup pipeline bug fix (mevcut vocab korunur)
• v1.1.0: yeni domain corpus eklendi, retrain ama vocab size aynı
• v2.0.0: vocab 32K → 50K — embedding incompat

12.2 Retraining schedule#

• Q1: feedback collect (HF issues, community feedback)
• Q2: corpus update (new Wikipedia dump, OSCAR refresh)
• Q3: retrain + evaluate
• Q4: release v(N+1).0.0 if vocab change, v(N).M.0 if backwards-compat

12.3 Community feedback channels#

• HuggingFace Hub repo Issues
• GitHub repo (training scripts hosted there)
• Blog post comments (sukruyusufkaya.com)
• Türkiye AI Discord/Slack

12.4 Deprecation policy#

12.5 Monitoring#

• HF download stats (haftalık check)
• HF model card star count
• Citing papers (Google Scholar alerts)
• GitHub clones / stars

12.6 Roadmap (v1.x, v2.x)#

• Augmented corpus: medical, legal sub-domains
• Improved fertility metric for compound words

• Multilingual variant (TR + EN co-training)
• Code-aware mode

• 50K vocab (larger context efficiency)
• Subword regularization training option
• Tools/agents integration tokens

Capstone Egzersizleri#

Egzersiz 1#

Egzersiz 2#

Egzersiz 3#

Egzersiz 4#

Egzersiz 5#

Egzersiz 6#

Egzersiz 7#

Egzersiz 8#

Egzersiz 9#

Egzersiz 10#

Modül 6 Envanteri (Tamamlandı)#

Genel Müfredat İlerleme#

Sıradaki Modül: Embedding Katmanı#