# Dataset Quality Pipeline: MinHash Dedupe + Perplexity Filter + Toxicity + Educational-Value

> Source: https://sukruyusufkaya.com/learn/fine-tuning-cookbook/ftc-dataset-quality-pipeline-minhash-perplexity
> Updated: 2026-08-13T20:44:14.489Z
> Category: Fine-Tuning Cookbook (Model-by-Model)
> Module: Part II — Tokenizer & Data Engineering
**TLDR:** Garbage in, garbage out. SFT dataset quality pipeline: MinHash LSH ile near-duplicate detection (~%30-40 duplicate olur), KenLM 5-gram perplexity filter (anlamsızları at), HateBERT-TR ile toxicity score, FineWeb-style educational-value scorer. RTX 4090'da 1M satır TR datasetini 25 dakikada temizleme.