Date: Mon, 2 Oct 2006 04:28:56 -0400
Reply-To: Jim Groeneveld <jim2stat@YAHOO.CO.UK>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Jim Groeneveld <jim2stat@YAHOO.CO.UK>
Subject: Re: Significance Testing with a Norms Database
Content-Type: text/plain; charset=ISO-8859-1
You may regard your 25 products as 25 control groups of 100 (or possibly as
one large control group of 25000). Then you may test your 26th product (cf.
an experimental group) with each of the 25 control groups (or with the whole
large group). I think an independent t-test may be valid (depending on type
of scale used to measure awareness) and different group frequencies do not
matter. However, you should note that, purely on the basis of coincidence,
about one (you don't know which one) out of 20 tests will 'prove'
significant, though it actually isn't. As the 25 tests are completely
independent you nevertheless don't need any adjustment to the alpha level
according to Keppel (Keppel, Geoffrey. 1991. Design and Analysis - A
Researcherís Handbook. Prentice Hall. Englewood Cliffs, New Jersey.)
But I understand you already have all your data and now you are thinking of
a statistical design. That is the world upside down, like buying a computer
and then thinking about what purpose to use it for. The only value your
(t-)tests may have in such an instance is explorative. Significance now is
nothing more than an indication, but not a 'prove'. Only if you had
hypotheses in advance and gathered data with the goal to test those
hypotheses you may have real significance, i.e. statistically 'proven' any
difference. Now you unfortunately can not show any significant difference at
Regards - Jim.
Jim Groeneveld, Netherlands
Statistician, SAS consultant
On Fri, 29 Sep 2006 12:03:43 -0700, Zai Saki <zaisaki@GMAIL.COM> wrote:
>I need help with understanding certain basic concepts of significance
>tests when comparing the results against a norms database.
>Our norms database has results on several metrics for 25 different
>products withing the same category. Each category has about 1000
>individual respondents rating the product. In total we have 25000
>observations on 25 products in our database. (Please note that each
>product is being rated by a different set of 1000 respondents).
>In conducting a new study for the 26th product, we want to stat test
>the metrics for the 26th product against the norms derived on the
>previous 25 products.
>1) What is the best approach to stat test such metrics. For example we
>want to stat test the average awareness of all 25 products to the
>average awareness of the 26th product.
>2) Are standard t-tests valid? If yes, is it appropriate to test 25000
>respondents against 1000 respondents.
>Any suggestions are greatly appreciated.