Pretraining Corpus

The pretraining corpus strongly determines what linguistic patterns, domain knowledge, and cultural structures a model will learn. Beyond data volume, diversity, cleanliness, licensing, and language distribution are all critical. The behavior of large models is often shaped by corpus character as much as by architecture. For that reason, data selection is an inseparable part of model design.