RE: [Corpora-List] Kappa coefficient for more than two annotators

From: Adam Kilgarriff (adam@lexmasterclass.com)
Date: Tue Jul 18 2006 - 19:29:47 MET DST

  • Next message: miles@inf.ed.ac.uk: "[Corpora-List] Research Position in Statistical Machine Translation"

    Narjes,

    Don't assume it will get you an informative result. Kappa depends on what a
    random guesser does, but the random guesser's results depend on the
    probability function you assume. It is easy to use kappa to spread more
    confusion that light. See section 5 of
    http://www.kilgarriff.co.uk/Publications/2003-K-EACLBuda-noburo.pdf

    Adam

    -----Original Message-----
    From: owner-corpora@lists.uib.no [mailto:owner-corpora@lists.uib.no] On
    Behalf Of Narjes Boufaden
    Sent: 18 July 2006 17:35
    To: corpora
    Subject: [Corpora-List] Kappa coefficient for more than two annotators

    Hi everybody,

    I'm looking for a perl script that estimates the Kappa coefficient for
    data coming from more than two annotators and with multiple classes.

    Thanks a lot for any help,

    Narjčs.

    This archive was generated by hypermail 2b29 : Tue Jul 18 2006 - 19:29:40 MET DST