Continual Pre-training TR: Catastrophic Forgetting Önleme + Replay Buffer

1. Catastrophic Forgetting — Niye?#

1. Replay buffer: EN data %10-15 mix
2. LR düşürme: Pre-train original'in 1/20'si (örn. 3e-5)
3. LoRA-style adapter: Base'i freeze, sadece adapter güncelle
4. Elastic weight consolidation (EWC): Önemli parametreleri korumak için L2 penalty

2. Replay Buffer Recipe#

from datasets import interleave_datasets

# Continual PT için mix
tr_data = load_dataset("tr-cleaned-50gb", streaming=True)
en_data = load_dataset("c4-en-sample", streaming=True)     # 10-20GB sample

mixed = interleave_datasets(
    [tr_data, en_data],
    probabilities=[0.88, 0.12],         # %12 EN replay
    seed=42,
    stopping_strategy="all_exhausted",
)

# Training
cfg = TrainingArguments(
    learning_rate=3e-5,                  # original 3e-4'ün 1/10'u
    warmup_ratio=0.005,                  # az warmup, ısınmış model
    weight_decay=0.1,                    # original kadar
    lr_scheduler_type="cosine",
    max_steps=125_000,                   # 2B token / 16K batch
    ...
)