Date: Fri, 3 Jun 2005 14:08:24 -0500
Reply-To: schmitta@UWM.EDU
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: "Thomas A. Schmitt" <schmitta@UWM.EDU>
Subject: Design/Analysis Questions #2
In-Reply-To: <OF25AB2D83.36E6D64A-ON88257014.00785AC3-88257014.0078E563@epamail.epa.gov>
Content-Type: text/plain; charset=ISO-8859-1
Hello All,
Here are some more details on my study. There are nine experimental conditions
and one control condition and examinee responses within each of these
conditions are simulated using the 1-PL and 3-PL models in IRT. Each examinee
(with a known theta) is given a 30-item test that is completed “normally” using
the probabilities generated from the models (this is the control condition).
The simulated CAT also included nine experimental conditions pertaining to
three different completion methods and the three different numbers of items
these methods were completed in. Thus, and examinee either guesses and does not
complete the last 3, 6, and 9 test items. For the guessing method examinees had
a 0%, 25% or 33% chance of answering an item correctly. These probabilities
replaced the model generated probabilities and were intended to simulate
speededness in the sense that examinees would have to hurry near the end of a
test to complete items. Thus, one control condition (normally completed 30-item
test) and 9 experimental conditions (3, 6, and 9 test items; 33%, 25%, and 0%)
were considered. My hypothesis is that as the number of items guessed on
increases (moving from 3 to 9 items) and the probability of getting these items
corrected decreased (moving from 33% to 0%) the examinees thetas will be
“significantly” underestimated.
I simulated each of the 200 examinees 100 times for the 9 experimental
conditions and then combined across replications.
Since this is a 3x3 within-subjects design I wanted to analyze the 9
experimental conditions appropriately in SAS. I’m still trying to decide how to
fit the control condition into the picture since it does not have the 2
factors. I would like to test the main effects and the interactions in SAS. One
of my concerns with this study has been statistical significance. Do I report
this or do I just simply use effects. Because of the small error and large
sample size the results will be very significant.
Since I have calculated Bias and RMSE for each of the 200 examinees one thought
here was to take a mean of the Bias and RMSE for each of the 200 examinees in
each of the 9 conditions along with the control condition and create a
histogram with the mean of the RMSEs and mean Biases. This would give me a mean
effect of sorts and allows me to see the increase in the underestimate of
theta. Does this seem reasonable even though the conditions are not separated?
Any thoughts on this design and data analyses and summarization and how to do
the 3x3 within-subjects design in SAS testing the main effects and interactions
would be appreciated. Anyway, I hope this clarifies somewhat my study.
Tom
| 
