Date:         Tue, 15 Jun 2004 13:03:32 -0300
Reply-To:     Hector Maletta <hmaletta@fibertel.com.ar>
Sender:       "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From:         Hector Maletta <hmaletta@fibertel.com.ar>
Subject:      Re: Comparing ranks
Content-Type: text/plain; charset="us-ascii"

Carmel:
Your colleague has indeed two different populations, composed of n in the
general public and m experts. Each subject in each sample is asked to rank
the tem items, so in fact you do not have two rankings, but m+n individual
rankings. Moreover, each kind is produced by different people, therefore you
cannot compute a Spearman correlation, which would require two ordinal
variables attributed to the same subjects. From individual rankings in the
first or second population you may also obtain a summary view of the
rankings. For instance, item A may have been put first by some subjects in
each population, second by others, yet third by others, etc. So you get a
DISTRIBUTION of positions for each item, within each population. This
distribution may be summarized in several ways to obtain a summary view of
the collective ranking, but these summaries require treating the ordering as
an interval, not an ordinal, variable. For instance, one may compute the
average position for each item within each population, and the standard
deviation of the distribution of positions around that average. Once this is
done in both populations, one may analyze the differences between both in
the average ranking of each item. Hope this helps

Hector



> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU]
> On Behalf Of Carmel McGinley
> Sent: Tuesday, June 15, 2004 11:59 AM
> To: SPSSX-L@LISTSERV.UGA.EDU
> Subject: Comparing ranks
>
>
> Dear List Members
>
>
>
> A colleague has asked me the following.  Initially, I thought
> of a spearman correlation coefficient or Mann Whitney test
> depending on how the data are structured but after more
> thought I am really not sure.  Is either appropriate or is
> there a better way?
>
>
>
> Regards
>
>
>
> Carmel
>
>
>
>
>
>
>
> Imagine as part of a phone survey you ask the public to rank
> 10 issues (referred to here as issues A - J) in order of how
> important those issues are to them.  Let's say they
> conveniently rank the 10 issues thus:
>
>
>
> 1st - issue A
>
> 2nd - issue B
>
> 3rd - issue C
>
> etc etc down to 10th, issue J
>
>
>
> Now imagine you put the same 10 issues to 'experts' and ask
> them to rank the same issues in order of how important they
> think the public consider those issues.  In other words, the
> experts are trying to guess the answers the public would have given.
>
>
>
> Let's say the experts get some of the order right, but some
> wrong, so they come up with something like this:
>
>
>
> 1st - A
>
> 2nd - C
>
> 3 - B
>
> 4 - E
>
> 5 - G
>
> 6 - F
>
> 7 - D
>
> 8 - I
>
> 9 - H
>
> 10 - J
>
>
>
> The eye tells you that this list correlates with the public's
> answers better than if the experts had just randomly ordered
> the 10 issues, although clearly they have made mistakes.
>
>
>
> What I want to know is whether there is some way of measuring
> how good the experts have performed.  What is the best and
> most commonly accepted measure to use?
>