Date: Thu, 23 Jun 2005 16:39:16 -0700
Reply-To: Dale McLerran <stringplayer_2@YAHOO.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Dale McLerran <stringplayer_2@YAHOO.COM>
Subject: Re: Concordant Discordant pair--What is it
Content-Type: text/plain; charset=iso-8859-1
--- "David L. Cassell" <cassell.david@EPAMAIL.EPA.GOV> wrote:
> "Nick ." <ni14@MAIL.COM> wrote back:
> > I was looking at Ian's and Chang's code more closely and both work
> > fine except for one big problem, which was perhaps my fault. Or
> > may still may not be a problem but here it goes:
> > In the example below I only made up a few pairs, i.e. I only put
> > down like 13 observations. In my real data set, I have close to 1
> > million observations, 970K of which have a y = 0 (RESPONSE = 0 No,
> > non-event) and about 23K have a y = 1 (RESPONSE = 1 Yes event).
> > I think Ian's code performs a cartesian product and I just used
> > Chang's code using only data steps. When I multiply 970K * 23K =
> > about 22 billion total pairs!!! My computer (Solaris box) is
> > out of space. I hope these codes (Ian's and Chang's) don't write
> > 22 billion records anywhere!!!
> Well, there's no getting around the fact that you need to do
> 22 billion pairwise comparisons, and you need to look at perhaps
> significantly more records in order to find all those pairs.
Hey, what gives here today. First I posted something that was
not a supportable response and now apparently David has caught the
same bug. There is a way to get around examining directly all
Ncase*Ncontrol combinations. See the code which I posted just a
bit ago. If we first sort the data by the predicted probability
of a success, then we can construct the number of concordant and
discordant pairs through an o(n) process. Of course, the sort
operation must be factored in. I believe that is an o(n*log(n))
process. Thus, we can improve on the efficiency of a SQL join
of all Ncase by Ncontrol combinations.
Fred Hutchinson Cancer Research Center
Ph: (206) 667-2926
Fax: (206) 667-5977
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around