# RLHF Klasik: Reward Model + PPO + KL Constraint — Niye Üretim Seti Terk Etti?

> Source: https://sukruyusufkaya.com/learn/fine-tuning-cookbook/ftc-rlhf-classical-reward-model-ppo-kl
> Updated: 2026-08-19T20:45:23.811Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part XI — Alignment & Preference Optimization
**TLDR:** RLHF (Christiano et al. 2017, InstructGPT 2022) — modern alignment'ın temeli. 3 aşama: SFT base + reward model train + PPO with KL constraint. Niye yarın ortada kayboldu? PPO'nun instability'si, value head'in maintenance burden'ı, DPO'nun pratik üstünlüğü. RTX 4090'da TRL ile mini-RLHF demo.