In the task Carletta discusses, you have a set J of judges, and each
judge classifies each of N items into one of M categories. For
example, J might be a set of doctors, N might be a set of patients,
and M might be a set of diagnoses. The kappa statistic is a way of
assessing the extent to which the doctors' diagnoses are consistent
with each other.
Now imagine the following situation. A hospital wants to make the
argument that nurses can diagnose patients as reliably as doctors. So
it takes the same set of N patients as described above, but instead of
a single group J of doctors doing the classification task, it has one
set J1 of doctors do the task (assigning each patient to one of M
categories) and also has a set J2 of nurses do the very same task,
i.e. take the same N patients and assign each to one of the same M
categories.
Clearly the kappa statistic can be used for each group of judges
separately, i.e. you can use it to assess how the nurses' within-group
reliability compares with the doctors' within-group reliability on the
same task. But that doesn't show that the nurses as a group can be
expected to generally make the *same* diagnoses. So the question is:
what is the proper statistical analysis for the hospital to use if it
wants to show that diagnoses made by nurses do or do not differ
significantly from those made by doctors?
Thanks,
Philip
----------------------------------------------------------------------------
Philip Resnik E-mail: philip.resnik@east.sun.com
Sun Microsystems Laboratories Work: (508) 442-0841
Two Elizabeth Drive Fax: (508) 250-5067
Chelmsford, MA 01824-4195 USA
----------------------------------------------------------------------------