|
I have 3 raters who are expereinced and 3 more who are less
experienced. They are all rating the same subjects. I would like to
compare the less to the more experienced raters by kappa scores and
have set the comparison up as follows:
combine the cells in a 2x2 format by group (exp and less exp), so that if
rater 2
+ -
+ 5 4
rater 1 - 6 25
rater 3
+ -
+ 3 6
rater 1 - 10 21
rater 2
+ -
+ 1 3
rater 3 - 4 32
i would combine to get -
rater
+ -
+ 9 13
rater - 20 78
I then do the same for the experienced readers and compare the kappa's
as suggested by Fleiss (p. 223) using the chi-sq statistic.
Is this legitimate? I'm concerned about the assumption of independence in
the combined 2x2 table as I've used the same rater twice in the combined
tabulations.
thanks for any direction>
Jeff
| 
