Date: Sun, 1 Jul 2007 20:59:33 +0000
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Ian Whitlock <iw1junk@COMCAST.NET>
Subject: Re: Help! Accessing observations from a dataset using a lookup
table (with constraint)
Summary: Read the problem
You wrote in part:
Howard, oh I gasped all right, my friend, I gasped BIG TIME! I guess
that the noise of the traffic on the Beltway between Rockville and
Washington, DC must have muted it out:-)
Actually, _BECAUSE_ of the amazing size of the MASTER data set, I ruled
out an in-memory solution such as Hash Tables or Formats. I didn't just
choose a SAS Index solution simply because I wrote the book. If the
prohibition against "modifying" MASTER applies even to building a SAS
index on ID, then I believe Alon is left standing alone without a valid
tool to use.
However, in the problem Alon <akadas@GMAIL.COM> explained that the big file
must be read sequentially, so an index solution is not available, and that
he needed only 300 records. It is easy to store 300 ID's in memory, read
(for some time), checking each ID to see if it is in the wanted ID's. So a
a partial in memory solution is most appropriate. Since there are many
ID's to check a hash would be the one most likely to complete.
I wonder how reliable the data on the file is and what sort of information
it contains. The answers may be worth a bigger gasp than the size.