Date:         Tue, 30 Aug 2011 15:45:49 +0000
Reply-To:     "Gerstle, John (CDC/OID/NCHHSTP)" <yzg9@CDC.GOV>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         "Gerstle, John (CDC/OID/NCHHSTP)" <yzg9@CDC.GOV>
Subject:      Re: Hash Object memory limitations
Comments: To: Andrew Tschirhart <andrew.tschirhart@GMAIL.COM>
In-Reply-To:  <201108292004.p7TAnQJK005820@wasabi.cc.uga.edu>
Content-Type: text/plain; charset="utf-8"

Andrew,
Another path to follow may be to re-strategize your process - why put the entire 5 GB dataset into a hash table?  Could you split it up into multiple steps? Multiple hash tables?  Also, check out Paul Dorfman's papers on Hash of hashes - using multiple hash tables when dealing with large datasets.

John Gerstle
Scientific Information Specialist
Centers for Disease Control and Prevention
NCHHSTP\DHAP-SE\QSDMB\Data Management Team
Phone: 404-639-3980
Fax: 404-639-8642
Email: yzg9 at cdc dot gov
Socrates, proclaimed: "I came to know one thing; that I know nothing".� �

"Every question I answer will simply lead to another question."

>>-----Original Message-----
>>From: owner-sas-l@listserv.uga.edu [mailto:owner-sas-l@listserv.uga.edu] On
>>Behalf Of Andrew Tschirhart
>>Sent: Monday, August 29, 2011 4:04 PM
>>To: SAS-L@LISTSERV.UGA.EDU
>>Subject: Hash Object memory limitations
>>
>>Hi all,
>>
>>I am running SAS 9.2 64-bit, version 2M2, on a Windows XP 64-bit machine
>>with 16 GB of memory.  I have a question about SAS hash tables.  It seems
>>that whenever I try to load a dataset bigger than 5 GB into a hash table,
>>the hash fails, and I get the standard "insufficient memory" error, i.e.:
>>
>>FATAL: Insufficient memory to execute data step program. Aborted during
>>the EXECUTION phase.
>>NOTE: The SAS System stopped processing this step because of insufficient
>>memory.”
>>
>> I am confused why this happens, because I have 16 GB of ram and am not
>>running any other large processes.  My MEMSIZE and REALMEMSIZE are both
>>set to 16GB, so it seems that SAS should be able to use all of my memory.
>>Additionally, when I load datasets in the 3GB-5GB range, I get a lot of
>>hard-drive thrashing, which makes me think that SAS has loaded some of the
>>table into virtual memory.
>>
>>At first, I thought that the issue might be that my compressed tables
>>were "growing" to become larger than 16 GB as they were uncompressed by
>>SAS.  However, even if I calcuate the table size based on the (record size
>>* number of records) formula, my tables still fall well short of the 16 GB
>>limitation
>>
>>My question is:
>>
>>Is there some other option I am unaware of that could be limiting my
>>memory usage?  For example, some sort of "memory per cpu" limitation on my
>>multiprocessor workstation?  Any way that I could test for this?
>>
>>Any help/thoughts would be greatly appreciated.  Thanks,
>>
>>Andrew T.