A software aimed primarily at doing G-studies exists. This is the GENOVA software. Probably acessable form the University of Iowa website, http://www.uiowa.edu/~itp/pages/SWGENOVA.SHTML.

GENOVA will probably give you all relevant results you need.

The basics of runnig GENOVA is discussed in Brennan, R. L. (2001). 'Generalizability theory' (New York: Springer-Verlag). also in his earlier monograph on G-theory (ACT, Iowa City, early 90s). I do not recall whether Shavelson and Webb discusses GENOVA in their book on G-theory , published in the early 90s.

Sigurgrimur Skulason Head of Dep. of Nationwide Testing Educational Testing Institute Su?urgotu 39 101 Reykjavik Iceland

tel. + 354 551 0560 fax. + 354 551 0590 www.namsmat.is

-----Original Message----- From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU]On Behalf Of Patricia Cleland Sent: Monday, March 24, 2003 2:25 PM To: SPSSX-L@LISTSERV.UGA.EDU Subject: Generalizability Study (G-study): Can SPSS handle this design?

In order to establish the rate of marker reliability, in the past we have randomly identified student work (booklets) and reinserted them for remarking. The scoring information from these reinserted booklets has been used to determine an estimate of inter-rater reliability or agreement. This estimate is then used to report scoring reliability to the public. However, because a marker does not necessarily mark multiple booklets in this process, rater variance is nested within student variance and is impossible to isolate.

This year we would to carry out a simple generalizability student to estimate the general marker, item and individual influences on students' scores as well as an overall generalizability coefficient.

Since we have about 200 markers marking about 140,000 students' booklets (for 3 subjects each in 2 grades), remarking all booklets is out of the question for fiscal reasons. The process that we are proposing is to randomly select one or more markers each day and reinsert their booklets as a group in to the marking process. Due to the way that booklets are handled during marking, this should mean that those booklets will be remarked by a small group of markers. For example, if the original marker marked 40 booklets, those booklets would be remarked by 4 markers each of whom marked about 10 booklets Thus, each group of booklets is marked by two identical raters which will allow us to isolate rater (marker) variance and student variance. By expanding this process to include multiple raters across multiple tasks, raters variance, item variance and student variance can be measured.

Is this an analysis that can be carried out using GLM-VARCOMP in SPSS11.5? If so, any suggestions for how to do this would be appreciated as I've never done this sort of analysis before.

If this is not doable in SPSS, does anyone know if it can be handled in SAS? (I, myself, don't 'speak' SAS but another member of the department does.)

Thanks, as always,

Pat