Date: Wed, 26 Mar 2003 23:05:19 -0000
Reply-To: Sigurgrimur Skulason <email@example.com>
Sender: "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From: Sigurgrimur Skulason <firstname.lastname@example.org>
Subject: Re: Generalizability Study (G-study): Can SPSS handle this design?
Content-Type: text/plain; charset="us-ascii"
A software aimed primarily at doing G-studies exists. This is the GENOVA
software. Probably acessable form the University of Iowa website,
GENOVA will probably give you all relevant results you need.
The basics of runnig GENOVA is discussed in Brennan, R. L. (2001).
'Generalizability theory' (New York: Springer-Verlag). also in his earlier
monograph on G-theory (ACT, Iowa City, early 90s). I do not recall whether
Shavelson and Webb discusses GENOVA in their book on G-theory , published in
the early 90s.
Head of Dep. of Nationwide Testing
Educational Testing Institute
tel. + 354 551 0560
fax. + 354 551 0590
From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU]On Behalf Of
Sent: Monday, March 24, 2003 2:25 PM
Subject: Generalizability Study (G-study): Can SPSS handle this design?
In order to establish the rate of marker reliability, in the past we have
randomly identified student work (booklets) and reinserted them for
remarking. The scoring information from these reinserted booklets has been
used to determine an estimate of inter-rater reliability or agreement. This
estimate is then used to report scoring reliability to the public. However,
because a marker does not necessarily mark multiple booklets in this
process, rater variance is nested within student variance and is impossible
This year we would to carry out a simple generalizability student to
estimate the general marker, item and individual influences on students'
scores as well as an overall generalizability coefficient.
Since we have about 200 markers marking about 140,000 students' booklets
(for 3 subjects each in 2 grades), remarking all booklets is out of the
question for fiscal reasons. The process that we are proposing is to
randomly select one or more markers each day and reinsert their booklets as
a group in to the marking process. Due to the way that booklets are handled
during marking, this should mean that those booklets will be remarked by a
small group of markers. For example, if the original marker marked 40
booklets, those booklets would be remarked by 4 markers each of whom marked
about 10 booklets Thus, each group of booklets is marked by two identical
raters which will allow us to isolate rater (marker) variance and student
variance. By expanding this process to include multiple raters across
multiple tasks, raters variance, item variance and student variance can be
Is this an analysis that can be carried out using GLM-VARCOMP in SPSS11.5?
If so, any suggestions for how to do this would be appreciated as I've never
done this sort of analysis before.
If this is not doable in SPSS, does anyone know if it can be handled in SAS?
(I, myself, don't 'speak' SAS but another member of the department does.)
Thanks, as always,