```Date: Wed, 12 Feb 1997 12:10:16 +0200 Reply-To: Dvora Zomer Sender: "SAS(r) Discussion" From: Dvora Zomer Subject: Re: BE CAREFUL! (Cohen (1960) "kappa" Thanks) Comments: To: Eric Seaberg In-Reply-To: <199702111726.MAA000.68@papaya.gsph.pitt.edu> Content-Type: TEXT/PLAIN; charset=US-ASCII Dear Eric: Your first example is due to your misunderstanding of what cappa is. The ususal (unweighted) cappa is equal to (SUM(p(i,j))-SUM(p(i,.)*p(.,j))/(1-SUM(p(i,.)*p(.,j))) i.e. the meaning of cappa is THE RELATIVE EXCESS of diagonal above expected value under independency, relatively to the maximum possible excess. For your example it is 0.539.... . Your second example is due to misunderstanding of the way, how proc FREQ make tables. In the case it will produce exactly the same matrix as in the first example, because it use the values (NONE, MILD,....) only as labels and use numbers of columns and rows (i,j) for all calculations. To get a full table you have to force FREQ to build a 4x4 matrix. options linesize=79; data a; input i j w; cards ; 1 2 10 2 1 0.0001 1 3 6 2 2 0.0001 3 4 0.0001 4 2 3 4 3 25 ; proc freq ; tables i*j /agree ; weight w; run; The SAS System 1 11:51 Wednesday, February 12, 1997 TABLE OF I BY J I J Frequency| Percent | Row Pct | Col Pct | 1| 2| 3| 4| Total ---------+--------+--------+--------+--------+ 1 | 0 | 10 | 6 | 0 | 16 | 0.00 | 22.73 | 13.64 | 0.00 | 36.36 | 0.00 | 62.50 | 37.50 | 0.00 | | 0.00 | 76.92 | 19.35 | 0.00 | ---------+--------+--------+--------+--------+ 2 | 0.0001 | 0.0001 | 0 | 0 | 0.0002 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 50.00 | 50.00 | 0.00 | 0.00 | | 100.00 | 0.00 | 0.00 | 0.00 | ---------+--------+--------+--------+--------+ 3 | 0 | 0 | 0 | 0.0001 | 0.0001 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 | | 0.00 | 0.00 | 0.00 | 100.00 | ---------+--------+--------+--------+--------+ 4 | 0 | 3 | 25 | 0 | 28 | 0.00 | 6.82 | 56.82 | 0.00 | 63.64 | 0.00 | 10.71 | 89.29 | 0.00 | | 0.00 | 23.08 | 80.65 | 0.00 | ---------+--------+--------+--------+--------+ Total 0.0001 13.0001 31 0.0001 44.0003 0.00 29.55 70.45 0.00 100.00 STATISTICS FOR TABLE OF I BY J Test of Symmetry ---------------- Statistic = 43.999 DF = 6 Prob = 0.001 Kappa Coefficients Statistic Value ASE 95% Confidence Bounds ------------------------------------------------------ Simple Kappa -0.000 0.000 -0.001 0.001 Weighted Kappa 0.166 0.043 0.081 0.251 Sample Size = 44.0003 All formula are presented in SAS/STAT Software Changes pp. 226-230. Yours, Ilya Novikov On Tue, 11 Feb 1997, Eric Seaberg wrote: > Be careful, know your data. The agree option in PROC FREQ does not always > return an appropriate kappa statistic. An example is shown below. > > > >Just wanted to thank the many people who responded to my initial query > >including: EVALENTE@aamc.org, lyonsb@ucs.orst.edu, > >TINAZZI@irfmn.mnegri.it, khardy@frosty.irss.unc.edu, lena@hevanet.com, > >schick@hrz.uni-marburg.de, sfbay0001@aol.com. Kappa can be easily obtained > >using SAS 6.09 or higher with the following script: > > > > proc freq; > > tables rater1*rater2/agree; > > run; > > > Suppose that 2 raters use the following system to rate 44 subjects: > > 0=none 1=mild 2=moderate 3=severe > > Example 1: > > Both rater 1 and rater 2 assign 'Mild' on 10 ratings > Rater 1 assigns 'Mild' and rater 2 assigns 'Moderate' on 3 ratings > Rater 1 assigns 'Moderate' and rater 2 assigns 'Mild' on 6 ratings > Both rater 1 and rater 2 assign 'Moderate' on 25 ratings > > The actual proportion of ratings with exact agreement > is (10+25)/(10+25+3+6)=.795 > > The kappa value returned by PROC FREQ is 0.540 > > ----------COMPARE THIS RESULT WITH THE FOLLOWING------------------ > > Example 2: > > Rater 1 assigns 'Mild' and rater 2 assigns 'None' on 10 ratings > Rater 1 assigns 'Mild' and rater 2 assigns 'Severe' on 3 ratings > Rater 1 assigns 'Moderate' and rater 2 assigns 'None' on 6 ratings > Rater 1 assigns 'Moderate' and rater 2 assigns 'Severe' on 25 ratings > > The actual proportion of ratings with exact agreement > is 0/(10+25+3+6)=0.000 > > The kappa value returned by PROC FREQ is 0.540 > > HOW CAN KAPPA BE 0.540 WHEN THE TWO RATERS NEVER AGREE? > > ----------------------------------------------------------------- > > The following code was used to calculate KAPPA for the above examples: > > ***EXAMPLE #1***; > data a; > r1=1; r2=1; freq=10; output; > r1=1; r2=2; freq=3; output; > r1=2; r2=1; freq=6; output; > r1=2; r2=2; freq=25; output; > run; > proc freq; > tables r1*r2/agree; > weight freq; > run; > > ***EXAMPLE #2***; > data a; > r1=1; r2=0; freq=10; output; > r1=1; r2=3; freq=3; output; > r1=2; r2=0; freq=6; output; > r1=2; r2=3; freq=25; output; > run; > proc freq; > tables r1*r2/agree; > weight freq; > run; > ----------------------------------------------------------------- > > > -Eric > > ********************************************************************* > * Eric Seaberg seaberg@edc1.gsph.pitt.edu * > * Dept. of Epidemiology phone: 412-624-5136 * > * Univ. of Pittsburgh fax: 412-624-3775 * > * Pittsburgh, PA http://www.pitt.edu/~epidecs/ * > ********************************************************************* > ```

Back to: Top of message | Previous page | Main SAS-L page