LISTSERV at the University of Georgia
Menubar Imagemap
Home Browse Manage Request Manuals Register
Previous messageNext messagePrevious in topicNext in topicPrevious by same authorNext by same authorPrevious page (March 2003)Back to main SPSSX-L pageJoin or leave SPSSX-L (or change settings)ReplyPost a new messageSearchProportional fontNon-proportional font
Date:   Wed, 26 Mar 2003 23:05:19 -0000
Reply-To:   Sigurgrimur Skulason <sigsk@namsmat.is>
Sender:   "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From:   Sigurgrimur Skulason <sigsk@namsmat.is>
Subject:   Re: Generalizability Study (G-study): Can SPSS handle this design?
Comments:   cc: Patricia Cleland <Patricia.Cleland@eqao.com>
In-Reply-To:   <se7ecf05.070@1du25-nfs-01.eqao.com>
Content-Type:   text/plain; charset="us-ascii"

A software aimed primarily at doing G-studies exists. This is the GENOVA software. Probably acessable form the University of Iowa website, http://www.uiowa.edu/~itp/pages/SWGENOVA.SHTML.

GENOVA will probably give you all relevant results you need.

The basics of runnig GENOVA is discussed in Brennan, R. L. (2001). 'Generalizability theory' (New York: Springer-Verlag). also in his earlier monograph on G-theory (ACT, Iowa City, early 90s). I do not recall whether Shavelson and Webb discusses GENOVA in their book on G-theory , published in the early 90s.

Sigurgrimur Skulason Head of Dep. of Nationwide Testing Educational Testing Institute Su?urgotu 39 101 Reykjavik Iceland

tel. + 354 551 0560 fax. + 354 551 0590 www.namsmat.is

-----Original Message----- From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU]On Behalf Of Patricia Cleland Sent: Monday, March 24, 2003 2:25 PM To: SPSSX-L@LISTSERV.UGA.EDU Subject: Generalizability Study (G-study): Can SPSS handle this design?

In order to establish the rate of marker reliability, in the past we have randomly identified student work (booklets) and reinserted them for remarking. The scoring information from these reinserted booklets has been used to determine an estimate of inter-rater reliability or agreement. This estimate is then used to report scoring reliability to the public. However, because a marker does not necessarily mark multiple booklets in this process, rater variance is nested within student variance and is impossible to isolate.

This year we would to carry out a simple generalizability student to estimate the general marker, item and individual influences on students' scores as well as an overall generalizability coefficient.

Since we have about 200 markers marking about 140,000 students' booklets (for 3 subjects each in 2 grades), remarking all booklets is out of the question for fiscal reasons. The process that we are proposing is to randomly select one or more markers each day and reinsert their booklets as a group in to the marking process. Due to the way that booklets are handled during marking, this should mean that those booklets will be remarked by a small group of markers. For example, if the original marker marked 40 booklets, those booklets would be remarked by 4 markers each of whom marked about 10 booklets Thus, each group of booklets is marked by two identical raters which will allow us to isolate rater (marker) variance and student variance. By expanding this process to include multiple raters across multiple tasks, raters variance, item variance and student variance can be measured.

Is this an analysis that can be carried out using GLM-VARCOMP in SPSS11.5? If so, any suggestions for how to do this would be appreciated as I've never done this sort of analysis before.

If this is not doable in SPSS, does anyone know if it can be handled in SAS? (I, myself, don't 'speak' SAS but another member of the department does.)

Thanks, as always,

Pat


Back to: Top of message | Previous page | Main SPSSX-L page