Date: Fri, 25 Feb 2005 14:03:06 -0300
Reply-To: Hector Maletta <hmaletta@fibertel.com.ar>
Sender: "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From: Hector Maletta <hmaletta@fibertel.com.ar>
Subject: Re: What is the difference between association tests and tests of
median differences?
In-Reply-To: <200502251512.j1PFCdpb019269@listserv.cc.uga.edu>
Content-Type: text/plain; charset="us-ascii"
Doug,
Just a few thoughts. You should distinguish between two aspects of your
question:
1. Is there a difference between groups in the population represented by my
sample?
2. If such a difference exists, is it large or small?
The first question is normally answered by observing the difference in the
sample, and asking whether the observed difference is "statistically
significant at the xx level, e.g. at the 95% level". Here, a positive answer
means that if the difference actually does not exist in the population,you
would get your results in less than 5% of all possible samples, and
therefore you reject the null hypothesis that the difference does not exist
in the population. In other words, you say that, given the observed
difference in the sample, it is unlikely that no difference exists in the
population.
Notice that the same difference would be deemed significant or non
significant depending on sample size. A large sample allows you to be more
confident with a small difference than you would be with a small sample.
This higher or lower confidence in your results depending on sample size is
totally independent of the actual sample result and the actual difference in
the population.
The second question is totally different, because it deals with substantive,
not statistical, significance or importance. Substantive significance
depends on your purposes, and the standards of comparison you may be using.
Suppose you have a 0.01% difference between two very large samples in the
value of a certain variable, and assume the samples are so large that in
fact that 0.01% difference is statistically significant (meaning that you
are pretty confident the difference also exists at population level). Now,
is that difference substantively important or unimportant? It depends on the
problem at hand. If that 0.01% is the difference between safe and poisonous
doses of a certain chemical, such a difference may be extremely important,
but if it is just a difference between, say, the average age of two groups
of workers, it may be utterly unimportant for most purposes.
The relation between the two is usually found in the notion of statistical
power, used to determine sample size depending on the size of the expected
effect or difference. When a small difference is important (and a small
difference in the difference is also important, as in the example with
poisonous chemicals), you will need to prove that the small difference found
in your sample is statistically significant, i.e. that such small difference
is actually likely to exist in the population, and thus you will need a
larger sample to be on the safe side. When you aim at proving only the
existence of a difference large or small, a smaller sample would be enough.
In your case, if the difference in the median response for two groups is
statistically significant, it would mean that you are pretty confident the
difference is not zero in the population. Statistical significance is about
the relationship between your sample and the population. Whether such
difference (the difference in the sample, or the inferred difference in the
population) is substantively important or not, will depend on the problem at
hand, i.e. in the expected difference in the consequences of such a
difference of opinion between the groups. If valuing a factor more, even
slightly more, relates to a specific behavior (such as buying one product
instead of another, or buying it more often) then the difference is
substantively important. Otherwise it is not, even if it may be
statistically significant.
Hector
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU]
> On Behalf Of Doug Roberts
> Sent: Friday, February 25, 2005 12:13 PM
> To: SPSSX-L@LISTSERV.UGA.EDU
> Subject: What is the difference between association tests and
> tests of median differences?
>
>
> I feel pretty stupid asking this but just can't get it
> straight in my mind. I have data where various groups ranked
> importance on a number of different factors (importance of a
> factor on their decision to do something - ranked on a Likert
> 5 point scale). I can't understand which test to use to
> answer my question of "is importance of a factor different
> among groups?" Also, is there another question I should be
> asking besides this?
>
> My understanding:
>
> Association test: tests whether there is an association
> between being in a certain group X (e.g., group 1, group 2,
> etc) and how important you rank the factor. I interpret
> significance results here to mean that members of a certain
> group X believe a factor is more (or less) important than
> members of another group X.
>
> Test of difference in median (or mean; I choose median
> because it was Likert scale data): tests whether the median
> importance of a factor to group X1 is different than the
> median importance of that factor to group X2. Significance
> results here I interpret to mean that group X1 believes a
> factor to be more (or less) important than group X2.
>
> Now see my problem? I intrepret both the same way. Are they
> the same? Can anyone shed some light on this?
>
> Thanks in advance, Doug
>
| 
