On Tue, 18 Jul 2006, Narjes Boufaden wrote:
> I'm looking for a perl script that estimates the Kappa
> coefficient for data coming from more than two annotators and
> with multiple classes.
I have such a script; I'll send it to you privately, as it's
probably not interesting to the entire list.
But I want to include a cautionary note, which should be of
interest to at least some members of the list: the term "kappa" is
ambiguous, as it refers to more than one coefficient. For two
coders, there is kappa (Cohen 1960) which calculates chance
agreement using individual coder marginals, and pi (Scott 1955)
which averages over these marginals; unfortunately pi is also often
called "kappa". For multiple coders, there is kappa (Fleiss 1971,
Siegel and Castellan 1988, Carletta 1996) which is actually a
generalization of Scott's pi; there's also another kappa (Davies
and Fleiss 1982) which is a generalization of Cohen's kappa...
These issues are explained in some detail DiEugenio and Glass 2004
(Computational Linguistics 30(1): 95-101), and in more detail in
Artstein and Poesio 2005 (currently under revision):
http://cswww.essex.ac.uk/technical-reports/2005/csm-437.pdf
Fortunately, you have more than two annotators, and indeed the
difference between the various coefficients called kappa decreases
as the number of coders grows (another Artstein and Poesio 2005):
http://privatewww.essex.ac.uk/~artstein/publications/bias-a5.pdf
The numerical differences between the variants of kappa are likely
to be not very large (especially if you have many annotators or
reach high agreement), but I feel it's important to report exactly
which coefficient is used.
-Ron.
This archive was generated by hypermail 2b29 : Wed Jul 19 2006 - 17:46:49 MET DST