Annotator agreement
Assess consistency between two human labelers assigning binary or multiclass categories.
Scikit-Learn · Function
cohen_kappa_score
Computes Cohen’s kappa, a chance-adjusted measure of agreement between two classification labelings. It supports optional ordinal disagreement weights, sample weights, explicit labels, and an undefined-result replacement value.
- Since
- 1.9
How it works
The function compares two label assignments sample by sample. It calculates observed agreement and adjusts it for the agreement expected from the label distributions; optional weights make disagreements ordinal, and sample weights change each observation's contribution.
Syntax
sklearn.metrics.cohen_kappa_score(y1, y2, *, labels=None, weights=None, sample_weight=None, replace_undefined_by=nan)Parameters
| Name | Type | Description |
|---|---|---|
y1 | array-like | First labeling or annotator’s target values. |
y2 | array-like | Second labeling or annotator’s target values. |
labels | array-like, default=None | Optional explicit set of labels included in the comparison. |
weights | {None, "linear", "quadratic"}, default=None | Optional disagreement weighting: None, "linear", or "quadratic". |
sample_weight | array-like, default=None | Optional per-sample weights. |
replace_undefined_by | float, default=np.nan | Value returned when the metric is undefined. |
Returns
float — The kappa statistic, a number between -1 and 1.
Key points
- Returns a chance-adjusted statistic for agreement between two annotators.
- The comparison is framed around two classification labelings.
- The numerator uses the empirical probability that the annotators assign the same label.
- The score corrects observed agreement using agreement expected under random labeling.
- Its upper bound represents complete agreement, while zero or lower indicates chance-level agreement.
Example: Weighted inter-annotator agreement
from sklearn.metrics import cohen_kappa_score
labeling1 = [2, 0, 2, 2, 0, 1]
labeling2 = [0, 0, 2, 2, 0, 2]
score = cohen_kappa_score(labeling1, labeling2, weights="quadratic")
print(score)Computes weighted agreement between two labelings; quadratic weighting gives greater penalty to larger category disagreements.
When to use it
Weighted ordinal ratings
Compare ordinal ratings where larger category gaps should count as more severe disagreements.
Weighted samples
Aggregate agreement when observations have unequal importance or sampling frequency.
Interview questions
QHow is Cohen’s kappa calculated?
It is a chance-adjusted agreement statistic. If \(p_o\) is observed agreement and \(p_e\) is agreement expected from random assignments, it computes \(\kappa=(p_o-p_e)/(1-p_e)\).
Follow-up: What does a negative result indicate? It indicates agreement below the level expected by chance; zero corresponds to chance-level agreement.
QWhat does the weights parameter do?
weights controls how disagreements are scored. With no weights, disagreements are treated equally; ordinal data can use linear or quadratic weighting to reflect category distance.
Follow-up: When would quadratic weighting be preferable? Use it when disagreements farther apart should receive substantially larger penalties than adjacent disagreements.
QWhich inputs and options control the comparison?
y1 and y2 are the two labelings being compared, and each sample position must correspond across them. labels can define the label set, while sample_weight gives individual observations different influence.
Follow-up: What is the purpose of replace_undefined_by? It specifies the value returned when the metric is undefined because its calculation requires division by zero.
Related
accuracy_scorebalanced_accuracy_scoreconfusion_matrixmatthews_corrcoef