```Date: Fri, 25 Feb 2005 14:03:06 -0300 Reply-To: Hector Maletta Sender: "SPSSX(r) Discussion" From: Hector Maletta Subject: Re: What is the difference between association tests and tests of median differences? Comments: To: Doug Roberts In-Reply-To: <200502251512.j1PFCdpb019269@listserv.cc.uga.edu> Content-Type: text/plain; charset="us-ascii" Doug, Just a few thoughts. You should distinguish between two aspects of your question: 1. Is there a difference between groups in the population represented by my sample? 2. If such a difference exists, is it large or small? The first question is normally answered by observing the difference in the sample, and asking whether the observed difference is "statistically significant at the xx level, e.g. at the 95% level". Here, a positive answer means that if the difference actually does not exist in the population,you would get your results in less than 5% of all possible samples, and therefore you reject the null hypothesis that the difference does not exist in the population. In other words, you say that, given the observed difference in the sample, it is unlikely that no difference exists in the population. Notice that the same difference would be deemed significant or non significant depending on sample size. A large sample allows you to be more confident with a small difference than you would be with a small sample. This higher or lower confidence in your results depending on sample size is totally independent of the actual sample result and the actual difference in the population. The second question is totally different, because it deals with substantive, not statistical, significance or importance. Substantive significance depends on your purposes, and the standards of comparison you may be using. Suppose you have a 0.01% difference between two very large samples in the value of a certain variable, and assume the samples are so large that in fact that 0.01% difference is statistically significant (meaning that you are pretty confident the difference also exists at population level). Now, is that difference substantively important or unimportant? It depends on the problem at hand. If that 0.01% is the difference between safe and poisonous doses of a certain chemical, such a difference may be extremely important, but if it is just a difference between, say, the average age of two groups of workers, it may be utterly unimportant for most purposes. The relation between the two is usually found in the notion of statistical power, used to determine sample size depending on the size of the expected effect or difference. When a small difference is important (and a small difference in the difference is also important, as in the example with poisonous chemicals), you will need to prove that the small difference found in your sample is statistically significant, i.e. that such small difference is actually likely to exist in the population, and thus you will need a larger sample to be on the safe side. When you aim at proving only the existence of a difference large or small, a smaller sample would be enough. In your case, if the difference in the median response for two groups is statistically significant, it would mean that you are pretty confident the difference is not zero in the population. Statistical significance is about the relationship between your sample and the population. Whether such difference (the difference in the sample, or the inferred difference in the population) is substantively important or not, will depend on the problem at hand, i.e. in the expected difference in the consequences of such a difference of opinion between the groups. If valuing a factor more, even slightly more, relates to a specific behavior (such as buying one product instead of another, or buying it more often) then the difference is substantively important. Otherwise it is not, even if it may be statistically significant. Hector > -----Original Message----- > From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] > On Behalf Of Doug Roberts > Sent: Friday, February 25, 2005 12:13 PM > To: SPSSX-L@LISTSERV.UGA.EDU > Subject: What is the difference between association tests and > tests of median differences? > > > I feel pretty stupid asking this but just can't get it > straight in my mind. I have data where various groups ranked > importance on a number of different factors (importance of a > factor on their decision to do something - ranked on a Likert > 5 point scale). I can't understand which test to use to > answer my question of "is importance of a factor different > among groups?" Also, is there another question I should be > asking besides this? > > My understanding: > > Association test: tests whether there is an association > between being in a certain group X (e.g., group 1, group 2, > etc) and how important you rank the factor. I interpret > significance results here to mean that members of a certain > group X believe a factor is more (or less) important than > members of another group X. > > Test of difference in median (or mean; I choose median > because it was Likert scale data): tests whether the median > importance of a factor to group X1 is different than the > median importance of that factor to group X2. Significance > results here I interpret to mean that group X1 believes a > factor to be more (or less) important than group X2. > > Now see my problem? I intrepret both the same way. Are they > the same? Can anyone shed some light on this? > > Thanks in advance, Doug > ```

Back to: Top of message | Previous page | Main SPSSX-L page