Inter-Annotator Agreement

Inter-annotator agreement is an important indicator used to assess labeling quality quantitatively. If different people assign similar labels to the same examples, the task definition and guideline are likely clear enough. Low agreement may indicate either ambiguous data or insufficiently clear labeling rules. Measures such as Cohen’s Kappa are commonly used for this purpose. This metric helps evaluate label quality not only through outputs, but through the reliability of the labeling process itself.