A measure of agreement between two raters, or between a classifier and the truth, that corrects for agreement expected by chance: κ = (p_o − p_e) / (1 − p_e). A value of 1 is perfect agreement and 0 is what chance alone would give.
Card 1 of 259. Answer: A measure of agreement between two raters, or between a classifier and the truth, that corrects for agreement expected by chance: κ = (p_o − p_e) / (1 − p_e). A value of 1 is perfect agreement and 0 is what chance alone would give.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Here p_o is the observed share of agreement and p_e the agreement expected if both raters labeled at random with their own label frequencies. Jacob Cohen introduced it in 1960. In data science it is used to check annotation quality before training on the labels, and as a classification metric that, unlike accuracy, isn't flattered by a dominant class.
Example: two annotators label 100 reviews as positive or negative and agree on 84. Annotator A says positive 70 times and B 60 times, so chance agreement is 0.7 × 0.6 + 0.3 × 0.4 = 0.54, and κ = (0.84 − 0.54) / (1 − 0.54) ≈ 0.65. Raw agreement of 84% sounds better than it is.
Follow-ups: verbal scales such as Landis and Koch's (0.61 to 0.80 is "substantial") are rules of thumb, and kappa depends on prevalence, so two raters with high raw agreement on a rare class can still get a low kappa. For ordinal labels, weighted kappa gives partial credit for near misses (quadratic weighted kappa is a common competition metric), and Fleiss' kappa extends the idea to more than two raters.
