van Oest, Rutger & Girard, Jeffrey M. (2021)

Weighting Schemes and Incomplete Data: A Generalized Bayesian Framework for Chance-Corrected Interrater Agreement

Psychological methods Doi: 10.1037/met0000412

Van Oest (2019) developed a framework to assess interrater agreement for nominal categories and complete data. We generalize this framework to all four situations of nominal or ordinal categories and complete or incomplete data. The mathematical solution yields a chance-corrected agreement coefficient that accommodates any weighting scheme for penalizing rater disagreements and any number of raters and categories. By incorporating Bayesian estimates of the category proportions, the generalized coefficient also captures situations in which raters classify only subsets of items; that is, incomplete data. Furthermore, this coefficient encompasses existing chance-corrected agreement coefficients: the S-coefficient, Scott’s pi, Fleiss’ kappa, and Van Oest’s uniform prior coefficient, all augmented with a weighting scheme and the option of incomplete data. We use simulation to compare these nested coefficients. The uniform prior coefficient tends to perform best, in particular, if one category has a much larger proportion than others. The gap with Scott’s pi and Fleiss’ kappa widens if the weighting scheme becomes more lenient to small disagreements and often if more item classifications are missing; missingness biases play a moderating role. The uniform prior coefficient often performs much better than the S-coefficient, but the S-coefficient sometimes performs best for small samples, missing data, and lenient weighting schemes. The generalized framework implies a new interpretation of chance-corrected weighted agreement coefficients: These coefficients estimate the probability that both raters in a pair assign an item to its correct category without guessing. Whereas Van Oest showed this interpretation for unweighted agreement, we generalize to weighted agreement.

van Oest, Rutger Daniel (2019)

Unconstrained Cholesky-based parametrization of correlation matrices

Communications in Statistics - Simulation and Computation Doi: 10.1080/03610918.2019.1628271 - Full text in research archive

Parameter estimation is relatively complicated for models containing correlation matrices, because the elements of correlation matrices are heavily constrained. We put forward a Cholesky-based parametrization that is easy to implement and allows for unconstrained parameter estimation. To compare the new parametrization with the commonly applied spherical parametrization, we use Monte Carlo simulation in which we estimate multivariate distributions containing Gaussian copulas. We show that the new parametrization performs well, in particular as the dimensionality of the multivariate distribution increases, computing times increase, and non-convergence occurs increasingly often.

van Oest, Rutger Daniel (2019)

A New Coefficient of Interrater Agreement: The Challenge of Highly Unequal Category Proportions

Psychological methods, 24(4), s. 439- 451. Doi: 10.1037/met0000183 - Full text in research archive

We derive a general structure that encompasses important coefficients of interrater agreement such as the S-coefficient, Cohen’s kappa, Scott’s pi, Fleiss’ kappa, Krippendorff’s alpha, and Gwet’s AC1. We show that these coefficients share the same set of assumptions about rater behavior; they only differ in how the unobserved category proportions are estimated. We incorporate Bayesian estimates of the category proportions and propose a new agreement coefficient with uniform prior beliefs. To correct for guessing in the process of item classification, the new coefficient emphasizes equal category probabilities if the observed frequencies are unstable due to a small sample, and the frequencies increasingly shape the coefficient as they become more stable. The proposed coefficient coincides with the S-coefficient for the hypothetical case of zero items; it converges to Scott’s pi, Fleiss’ kappa, and Krippendorff’s alpha as the number of items increases. We use simulation to show that the proposed coefficient is as good as extant coefficients if the category proportions are equal and that it performs better if the category proportions are substantially unequal.

Andreassen, Tor W.; van Oest, Rutger Daniel & Lervik-Olsen, Line (2018)

Customer Inconvenience and Price Compensation: A Multiperiod Approach to Labor-Automation Trade-Offs in Services

Journal of Service Research, 21(2), s. 173- 183. Doi: 10.1177/1094670517738370

van Oest, Rutger Daniel & Knox, George (2015)

Håndtering av kundeklager : hva kan vi lære av faktisk kundeatferd?

Magma - Tidsskrift for økonomi og ledelse, 18(4), s. 72- 74.

Koulaei, Afra; van Oest, Rutger Daniel & Bharadwaj, Sundar (2019)

Familiarity Mitigates the Effect of Team Reward on Functional Conflict and Citizenship Behavior

[Academic lecture]. Academy of Management Annual Meeting.

Ungureanu, Delia Olga & van Oest, Rutger Daniel (2018)

When Customer Spending Backfires: Customer Trade-offs Between Durable Products Replacements and Add-on Purchases

[Academic lecture]. EMAC Conference.

Ungureanu, Delia Olga & van Oest, Rutger Daniel (2018)

When Customer Spending Backfires: Customer Trade-offs Between Durable Products Replacements and Add-on Purchases

[Academic lecture]. Marketing Science Conference.

van Oest, Rutger Daniel & Knox, George (2018)

Incorporating Satisfaction Surveys Into Customer Base Analysis

[Academic lecture]. EMAC 2018.

van Oest, Rutger Daniel (2017)

Inter-rater Reliability: The Challenge of Highly Unequal Category Sizes

[Academic lecture]. 2017 Marketing Science Conference.