Date: Sun, 29 Jan 2006 07:46:56 -0500
Reply-To: "Richard A. DeVenezia" <rdevenezia@WILDBLUE.NET>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: "Richard A. DeVenezia" <rdevenezia@WILDBLUE.NET>
Subject: Re: What is maximum number of variables allowed
David L Cassell wrote:
> art297@NETSCAPE.NET replied:
>> Then I was even more surprised that 2 million worked after having
>> failed previously.
>>
>> Then, I was totally confused when SAS9 failed with 2097150 and
>> returned the message that the maximum allowed for the file was
>> 16777191.
>>
>> I would think that someone from SAS needs to respond regarding what
>> the actual number is.
>
> I don't think there *is* a fixed number.
>
> I think that your work, as well as other experiments around here, are
> demonstrating
> that this is memory-dependent. Which could mean that - as your OS
> adjusts the
> amount of memory available to the SAS system - you may continue to get
> fluctuating
> results.
>
> Of course, I'm probably wrong. But it seems clear that the number of
> variables
> (and the length of the PDV created by that number of variables) is
> going to be
> memory-dependent.
>
> But larger than we usually work with...
>
> David
16777191
Interestingly 16,777,191 is 2**24.
A row with 2**24 numeric variables (double precision reals) of length 8
requires 2**27 bytes of storage/working space, plus overhead and alignment
issues (if any)
One speculation is that internally 32-bit positive integers (31 if signed)
are used to index into the memory buffer of a row, or at least index into a
hash mapping to the address of a variable -- this would indicate the
maximum allocation of a rows internal buffer is 4GB (2**32) or 2GB (2**31).
There is a 2**4 factor ratio between 2**31 and 2**27. This might indicate
the internal scheme for dealing with a single wide row is perhaps utilizing
a paging algorithm that requires adding 16 bytes to each variable to make
it's value locatable in the working space.
Richard A. DeVenezia
http://www.devenezia.com/
