[Corpora-List] Results of the query: Kappa coefficient for more than two annotators

From: Narjes Boufaden (boufaden@iro.umontreal.ca)
Date: Wed Jul 19 2006 - 17:30:18 MET DST

  • Next message: Hsin-Hsi Chen: "[Corpora-List] Call for Participation: Opinion Analysis Task at NTCIR-6"

    Hi everybody,

    Thank you for all of you who answered my question. Here are the list of
    your suggestions:

    + Adam Kilgarriff pointed out some of the problems related to the
    assumptions made when computing kappa.

    + Norton Roman suggested the use of Krippendorf's alpha to evaluate
    annotator agreement.

    + Ron Artstein kindly sent a script for the kappa (multiple juges). You
    ca ask him directly to have the script.

    + Andrei Popescu-Belis proposed a website where is available an
    implementation (MS-DOS) of the Fleiss approcah (an extention of the
    Cohen's kappa coefficient for multiple juges) : http://kappa.chez-alice.fr

    Thanks again for all your suggestions.

    Best regards, Narjčs.

    This archive was generated by hypermail 2b29 : Wed Jul 19 2006 - 17:30:20 MET DST