Are you saying that two people had opposite ratings on the same construct? A zero or close to zero correlation would mean lack of agreement. A negative correlation of any magnitude would mean that the raters opposed each other? Did you try crosstabs or scatter plots of rating pairs, 3d scatter plots of rating triplets?

If there is clearly an apparent negative correlation between two ratings of the same stimuli on the same construct, I would definitely go back and see if the data were entered correctly.

Art Art@DrKendall.org Social Research Consultants University Park, MD USA (301) 864-5570

Michael Jerin wrote:

>Paul, > >Thanks for responding. The problem I raised was one of assessing inter-rater reliability relative to mixed sleep apnea. I had a small sample size ( n = 16), and used a >two way mixed effect model with a consistency definition (actually I tried combinations of all the available options after getting this result with virtually the same outcome). A negative correlation is possible under these circumstances (when a lack of agreement is present among the raters), what I cannot understand is the correlation that exceeds negative 1.00. While I agree the data could be the source of the problem here, if you have any other thoughts I'd appreciate hearing them. > >Thanks again! > >mike > > >----- Original message ----------------------------------------> >From: Paul R Swank <Paul.R.Swank@uth.tmc.edu> >To: <drj48@earthlink.net>, Spss <SPSSX-L@LISTSERV.UGA.EDU> >Received: Thu, 13 May 2004 18:14:20 -0500 >Subject: RE: Problems with intra-class correlation > > > >>When I have seen it (using Cronbach's alpha, an intraclass correlation >>coefficient) it meant a really bad model fit. As I recall, somes items were >>negatively correlated with with others (it was a reliability analysis of a >>short scale). Since alpha assumes all items measure the same construct, then >>if they don't, anything can happen. >> >> > > > >>Paul R. Swank, Ph.D. >>Professor, Developmental Pediatrics >>Medical School >>UT Health Science Center at Houston >> >> > > > > >>-----Original Message----- >>From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] On Behalf Of >>Michael Jerin >>Sent: Thursday, May 13, 2004 9:54 AM >>To: SPSSX-L@LISTSERV.UGA.EDU >>Subject: Problems with intra-class correlation >> >> > > > > >>Dear SPSSX list, >> >> > > > >>I'm brand new to the list, and so beg your forgiveness if I unintentionally >>violate any list norms or if this question is irrelevant. >> >> > > > >>In a recent analysis of some data using the intraclass correlation procedure >>(two way mixed effect model, consistency definition), I obtained a 95% >>confidence interval of -2.020 -- 0.631. The lower limit exceeds a value of >>-1.00, has anyone ever experienced this type of anomaly? This was the only >>problem in a series of comparisons that were ran. All of the upper limits >>were below 1.00. Is this a quirk of SPSS or a problem with the data? >> >> > > > >>Any insight would be appreciated. If this isn't a question for the full >>list, please feel free to respond to me personally. >> >> > > > >>Thanks! >> >> > > > >>Mike Jerin >>St. Ambrose University >>drj48@earthlink.net >> >> > > >