Date: Wed, 12 Feb 1997 12:10:16 +0200
Reply-To: Dvora Zomer <epid04@post.tau.ac.il>
Sender: "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From: Dvora Zomer <epid04@POST.TAU.AC.IL>
Subject: Re: BE CAREFUL! (Cohen (1960) "kappa" Thanks)
In-Reply-To: <199702111726.MAA000.68@papaya.gsph.pitt.edu>
Content-Type: TEXT/PLAIN; charset=US-ASCII
Dear Eric:
Your first example is due to your misunderstanding of what cappa is.
The ususal (unweighted) cappa is equal to
(SUM(p(i,j))-SUM(p(i,.)*p(.,j))/(1-SUM(p(i,.)*p(.,j)))
i.e. the meaning of cappa is THE RELATIVE EXCESS of diagonal above
expected value under independency, relatively to the maximum possible
excess. For your example it is 0.539.... .
Your second example is due to misunderstanding of the way, how proc FREQ
make tables. In the case it will produce exactly the same matrix as in the
first example, because it use the values (NONE, MILD,....) only as labels
and use numbers of columns and rows (i,j) for all calculations. To get a
full table you have to force FREQ to build a 4x4 matrix.
options linesize=79;
data a;
input i j w;
cards ;
1 2 10
2 1 0.0001
1 3 6
2 2 0.0001
3 4 0.0001
4 2 3
4 3 25
;
proc freq ;
tables i*j /agree ;
weight w;
run;
The SAS System 1
11:51 Wednesday, February 12, 1997
TABLE OF I BY J
I J
Frequency|
Percent |
Row Pct |
Col Pct | 1| 2| 3| 4| Total
---------+--------+--------+--------+--------+
1 | 0 | 10 | 6 | 0 | 16
| 0.00 | 22.73 | 13.64 | 0.00 | 36.36
| 0.00 | 62.50 | 37.50 | 0.00 |
| 0.00 | 76.92 | 19.35 | 0.00 |
---------+--------+--------+--------+--------+
2 | 0.0001 | 0.0001 | 0 | 0 | 0.0002
| 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| 50.00 | 50.00 | 0.00 | 0.00 |
| 100.00 | 0.00 | 0.00 | 0.00 |
---------+--------+--------+--------+--------+
3 | 0 | 0 | 0 | 0.0001 | 0.0001
| 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| 0.00 | 0.00 | 0.00 | 100.00 |
| 0.00 | 0.00 | 0.00 | 100.00 |
---------+--------+--------+--------+--------+
4 | 0 | 3 | 25 | 0 | 28
| 0.00 | 6.82 | 56.82 | 0.00 | 63.64
| 0.00 | 10.71 | 89.29 | 0.00 |
| 0.00 | 23.08 | 80.65 | 0.00 |
---------+--------+--------+--------+--------+
Total 0.0001 13.0001 31 0.0001 44.0003
0.00 29.55 70.45 0.00 100.00
STATISTICS FOR TABLE OF I BY J
Test of Symmetry
----------------
Statistic = 43.999 DF = 6 Prob = 0.001
Kappa Coefficients
Statistic Value ASE 95% Confidence Bounds
------------------------------------------------------
Simple Kappa -0.000 0.000 -0.001 0.001
Weighted Kappa 0.166 0.043 0.081 0.251
Sample Size = 44.0003
All formula are presented in SAS/STAT Software Changes pp. 226-230.
Yours,
Ilya Novikov
On Tue, 11 Feb 1997, Eric Seaberg wrote:
> Be careful, know your data. The agree option in PROC FREQ does not always
> return an appropriate kappa statistic. An example is shown below.
>
>
> >Just wanted to thank the many people who responded to my initial query
> >including: EVALENTE@aamc.org, lyonsb@ucs.orst.edu,
> >TINAZZI@irfmn.mnegri.it, khardy@frosty.irss.unc.edu, lena@hevanet.com,
> >schick@hrz.uni-marburg.de, sfbay0001@aol.com. Kappa can be easily obtained
> >using SAS 6.09 or higher with the following script:
> >
> > proc freq;
> > tables rater1*rater2/agree;
> > run;
>
>
> Suppose that 2 raters use the following system to rate 44 subjects:
>
> 0=none 1=mild 2=moderate 3=severe
>
> Example 1:
>
> Both rater 1 and rater 2 assign 'Mild' on 10 ratings
> Rater 1 assigns 'Mild' and rater 2 assigns 'Moderate' on 3 ratings
> Rater 1 assigns 'Moderate' and rater 2 assigns 'Mild' on 6 ratings
> Both rater 1 and rater 2 assign 'Moderate' on 25 ratings
>
> The actual proportion of ratings with exact agreement
> is (10+25)/(10+25+3+6)=.795
>
> The kappa value returned by PROC FREQ is 0.540
>
> ----------COMPARE THIS RESULT WITH THE FOLLOWING------------------
>
> Example 2:
>
> Rater 1 assigns 'Mild' and rater 2 assigns 'None' on 10 ratings
> Rater 1 assigns 'Mild' and rater 2 assigns 'Severe' on 3 ratings
> Rater 1 assigns 'Moderate' and rater 2 assigns 'None' on 6 ratings
> Rater 1 assigns 'Moderate' and rater 2 assigns 'Severe' on 25 ratings
>
> The actual proportion of ratings with exact agreement
> is 0/(10+25+3+6)=0.000
>
> The kappa value returned by PROC FREQ is 0.540
>
> HOW CAN KAPPA BE 0.540 WHEN THE TWO RATERS NEVER AGREE?
>
> -----------------------------------------------------------------
>
> The following code was used to calculate KAPPA for the above examples:
>
> ***EXAMPLE #1***;
> data a;
> r1=1; r2=1; freq=10; output;
> r1=1; r2=2; freq=3; output;
> r1=2; r2=1; freq=6; output;
> r1=2; r2=2; freq=25; output;
> run;
> proc freq;
> tables r1*r2/agree;
> weight freq;
> run;
>
> ***EXAMPLE #2***;
> data a;
> r1=1; r2=0; freq=10; output;
> r1=1; r2=3; freq=3; output;
> r1=2; r2=0; freq=6; output;
> r1=2; r2=3; freq=25; output;
> run;
> proc freq;
> tables r1*r2/agree;
> weight freq;
> run;
> -----------------------------------------------------------------
>
>
> -Eric
>
> *********************************************************************
> * Eric Seaberg seaberg@edc1.gsph.pitt.edu *
> * Dept. of Epidemiology phone: 412-624-5136 *
> * Univ. of Pittsburgh fax: 412-624-3775 *
> * Pittsburgh, PA http://www.pitt.edu/~epidecs/ *
> *********************************************************************
>
