Dear Max

One way to get a significance test would be bootstrapping. I'm not sure where I came across the idea, but it's straightforward in principle. On each of say 500 bootstrap replications, you randomize the group membership eg by randomly reordering the group membership variable, but keeping the original order of the rest of the data. You calculate rwg on each replication, and store the rwg value. This gives you a distribution for rwg against which you can test the rwg value calculated with the proper group membership. If it is within the top or bottom 2.5% of the bootstrap distribution, you have significance.

This method was proposed in a paper, but I can't remember where I saw it. It would certainly be possible to implement it with SPSS syntax.

Regards Garry

-----Original Message----- From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] On Behalf Of Max Freund Sent: 24 November 2008 00:29 To: SPSSX-L@LISTSERV.UGA.EDU Subject: Re: SPSS syntax for r*WG(J) and r'WG(J)

Thanks Gary.

Actually, the LeBreton & Senter article is where I got the syntax I've been using for rWG(J), but I don't think it includes a significance test. They basically just give some thumbnail cutoffs.

My problem is that a number of the results have come up as negative numbers (indicating bimodal or other odd distributions of ratings). I've been counting those as 0, but I wanted to try the r* or r' variants to see if they deal better with my data.

(BTW, if anyone else is looking to do multi-level analysis, the LeBreton & Senter article is a great overview of inter-rater reliability and agreement stats.)

Anyone else have any suggestions?

Best, Max

-- Max Freund, M.I.I.M. . max.freund@cgu.edu . (909) 632-1624 Doctoral Student in Organizational Behavior, Claremont Graduate University (www.cgu.edu/sbos) Partner, LF Leadership (www.lfleadership.com)

On Nov 23, 2008, at 6:13 AM, Garry Gelade wrote:

> Max, > > You can find SPSS syntax for rwG(J) here, which may help (but I don't > know where you can find syntax for the r* and r' variants.) > > LeBreton, J. M. & Senter,J. L. (2008). Answers to 20 Questions About > Interrater Reliability and Interrater Agreement. Organizational > Research Methods,11, p 815-852. > > Regards > Garry Gelade > Business Analytic Ltd. > > > -----Original Message----- > From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] On Behalf > Of Max Freund > Sent: 23 November 2008 11:59 > To: SPSSX-L@LISTSERV.UGA.EDU > Subject: SPSS syntax for r*WG(J) and r'WG(J) > > Hello all, > > Does anyone happen to have SPSS syntax for calculating inter-rater > agreement using r*WG(J) or r'WG(J)? > > I'm not totally clear if there's a significance test for these stats > (some form of chi-square?) or if it's just a judgment call, but if > there is a significance testing syntax for them (and for rWG(J)), I'd > love to get that too! > > Here are references: > > r*WG: > Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A Revised > Index of Interrater Agreement for Multi-Item Ratings of a Single > Target. > Applied > Psychological Measurement, 23(2), 127-135. > > r'WG: > Lindell, M. K. (2001). Assessing and Testing Interrater Agreement on a > Single Target Using Multi-Item Rating Scales. Applied Psychological > Measurement, 25(1), 89-99. > > rWG: > James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within- > group interrater reliability with and without response bias. Journal > of Applied Psychology, 69(1), 85-98. > James, L. R., Demaree, R. G., & Wolf, G. (1993). r-sub(wg): An > assessment of within-group interrater agreement. Journal of Applied > Psychology, 78(2), 306-309. > > Thanks, > Max Freund > > > -- > Max Freund, M.I.I.M. . max.freund@cgu.edu . (909) 632-1624 Doctoral > Student in Organizational Behavior, Claremont Graduate University > (www.cgu.edu/sbos) Partner, LF Leadership (www.lfleadership.com) > > ======= > To manage your subscription to SPSSX-L, send a message to > LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except > the command. To leave the list, send the command SIGNOFF SPSSX-L For a > list of commands to manage subscriptions, send the command INFO > REFCARD > > __________ NOD32 3632 (20081121) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except > the command. To leave the list, send the command SIGNOFF SPSSX-L For a > list of commands to manage subscriptions, send the command INFO > REFCARD

======= To manage your subscription to SPSSX-L, send a message to LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

__________ NOD32 3632 (20081121) Information __________

This message was checked by NOD32 antivirus system. http://www.eset.com

===================== To manage your subscription to SPSSX-L, send a message to LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD