Exposure Bias

The problem in which a model trained on correct past context must face its own imperfect history during inference.

Exposure bias refers to the mismatch between training and inference conditions, especially in autoregressive generative models. A model trained with correct past tokens may, during actual use, be exposed to its own imperfect predictions. This can cause small errors to accumulate over time and degrade generation quality. It shows why in sequence modeling, not only architecture but also training strategy matters deeply.