van Oest, Rutger Daniel (2018)

A New Coefficient of Interrater Agreement: The Challenge of Highly Unequal Category Proportions

Psychological methods Doi: 10.1037/met0000183 - Full text in research archive

We derive a general structure that encompasses important coefficients of interrater agreement such as the S-coefficient, Cohen’s kappa, Scott’s pi, Fleiss’ kappa, Krippendorff’s alpha, and Gwet’s AC1. We show that these coefficients share the same set of assumptions about rater behavior; they only differ in how the unobserved category proportions are estimated. We incorporate Bayesian estimates of the category proportions and propose a new agreement coefficient with uniform prior beliefs. To correct for guessing in the process of item classification, the new coefficient emphasizes equal category probabilities if the observed frequencies are unstable due to a small sample, and the frequencies increasingly shape the coefficient as they become more stable. The proposed coefficient coincides with the S-coefficient for the hypothetical case of zero items; it converges to Scott’s pi, Fleiss’ kappa, and Krippendorff’s alpha as the number of items increases. We use simulation to show that the proposed coefficient is as good as extant coefficients if the category proportions are equal and that it performs better if the category proportions are substantially unequal.

Andreassen, Tor W.; van Oest, Rutger Daniel & Lervik-Olsen, Line (2018)

Customer Inconvenience and Price Compensation: A Multiperiod Approach to Labor-Automation Trade-Offs in Services

Journal of Service Research, 21(2), s. 173- 183. Doi: 10.1177/1094670517738370

van Oest, Rutger Daniel & Knox, George (2015)

Håndtering av kundeklager : hva kan vi lære av faktisk kundeatferd?

Magma - Tidsskrift for økonomi og ledelse, 18(4), s. 72- 74.

Knox, George & van Oest, Rutger Daniel (2014)

Customer Complaints and Recovery Effectiveness: A Customer Base Approach

Journal of marketing, 78(5), s. 42- 57. Doi: 10.1509/jm.12.0317

van Oest, Rutger Daniel (2013)

Why are Consumers Less Loss Averse in Internal than External Reference Prices?

Journal of Retailing, 89(1), s. 62- 71. Doi: 10.1016/j.jretai.2012.08.003

Ungureanu, Delia Olga & van Oest, Rutger Daniel (2018)

When Customer Spending Backfires: Customer Trade-offs Between Durable Products Replacements and Add-on Purchases

[Academic lecture]. EMAC Conference.

Ungureanu, Delia Olga & van Oest, Rutger Daniel (2018)

When Customer Spending Backfires: Customer Trade-offs Between Durable Products Replacements and Add-on Purchases

[Academic lecture]. Marketing Science Conference.

van Oest, Rutger Daniel & Knox, George (2018)

Incorporating Satisfaction Surveys Into Customer Base Analysis

[Academic lecture]. EMAC 2018.

van Oest, Rutger Daniel (2017)

Inter-rater Reliability: The Challenge of Highly Unequal Category Sizes

[Academic lecture]. 2017 Marketing Science Conference.

Ungureanu, Delia Olga & van Oest, Rutger Daniel (2016)

The Role of Customer Satisfaction and Acquisition Channel in Incentivized Referral Programs

[Academic lecture]. EMAC Conference.