Geography

A call to interpret disagreement components during classification assessment

Document Type

Article

Abstract

This foresight manuscript proposes several ideas concerning how to conduct insightful classification assessment. Authors should report disagreement components that relate to the research question, without anointing results as acceptable or good. This manuscript reviews the citations of the 2011 manuscript entitled ‘Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment’, which gave two recommendations: (1) do not use Kappa and (2) use disagreement components. We analyzed 200 articles that cited the Death to Kappa manuscript. A quarter of the articles followed both recommendations, another quarter followed only the first recommendation, another quarter followed only the second recommendation, and the last quarter followed neither recommendation. The attempt to replace Kappa with disagreement components has been partially effective, while Kappa continues to haunt several professions. We discuss misguided uses of Percent Correct and Kappa in Remote Sensing and Land Change Modeling. The concepts are general and thus relate to additional fields. Authors frequently use arbitrary thresholds of metrics to claim that results are acceptable. However, the notion that results can be acceptable or not is inherently unscientific. Scientists must use a metric that addresses a clear research question in which the scientists have no vested interest in the results.

Publication Title

International Journal of Geographical Information Science

Publication Date

3-2025

Volume

14

Issue

4

ISSN

1365-8816

DOI

10.1080/13658816.2025.2469830

Keywords

accuracy, error, model

Link to Full Text

Share

COinS