LISTSERV at the University of Georgia
Menubar Imagemap
Home Browse Manage Request Manuals Register
Previous messageNext messagePrevious in topicNext in topicPrevious by same authorNext by same authorPrevious page (May 2006)Back to main SPSSX-L pageJoin or leave SPSSX-L (or change settings)ReplyPost a new messageSearchProportional fontNon-proportional font
Date:   Tue, 23 May 2006 11:55:43 -0400
Reply-To:   Richard Ristow <wrristow@mindspring.com>
Sender:   "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From:   Richard Ristow <wrristow@mindspring.com>
Subject:   Re: about limits in spss
Comments:   To: rbriceno@sanigest.com
In-Reply-To:   <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAkGkV+lGCG0m RKWVcpp9WCsKAAAAQAAAAP/rSMZpZnUG57boYfXCn8QEAAAAA@sanigest.com>
Content-Type:   text/plain; charset=us-ascii; format=flowed; x-avg-checked=avg-ok-18CE4DD3

At 11:24 AM 5/23/2006, Rodrigo Briceno wrote:

>Dear listers. I would like to know where I can find information about >the maximum number of variables handled by spss, the maximum number of >observations, and technical specifications like these.

This is along the lines of a FAQ, so here we go. The below is by Jon Peck of SPSS, Incl, and applies to all recent versions of SPSS. Additional points:

. For most operations, increasing the number of cases will increase the running time about in proportion

. Increasing the number of variables will generally increase the running time about in proportion, even if you're not using them all, because the running time is dominated by the time to read the file from disk, i.e. the total file size

. After some point hard to estimate (though larger if the machine has more RAM), increasing the number of variables will increase the running out of all proportion, because it's hard to put one case in RAM without paging.

At 10:25 AM 6/5/2003, Peck, Jon [of SPSS, Inc.] wrote:

>There are several points to making regarding very wide files and huge >datasets. > >First, the theoretical SPSS limits are > >Number of variables: (2**31) -1 >Number of cases: (2**31) - 1 > >In calculating these limits, count one for each 8 bytes or part >thereof of a string variable. An a10 variable counts as two >variables, for example. > >Approaching the theoretical limit on the number of variables, however, >is a very bad idea in practice for several reasons. > >1. These are the theoretical limits in that you absolutely cannot go >beyond them. But there are other environmentally imposed limits that >you will surely hit first. For example, Windows applications are >absolutely limited to 2GB of addressable memory, and 1GB is a more >practical limit. Each dictionary entry requires about 100 bytes of >memory, because in addition to the variable name, other variable >properties also have to be stored. (On non-Windows platforms, SPSS >Server could, of course, face different environmental >limits.) Numerical variable values take 8 bytes as they are held as >double precision floating point values. > >2. The overhead of reading and writing extremely wide cases when you >are doubtless not using more than a small fraction of them will limit >performance. And you don't want to be paging the variable >dictionary. If you have lots of RAM, you can probably reach between >32,000 and 100,000 variables before memory paging degrades performance >seriously. > >3. Dialog boxes cannot display very large variable lists. You can use >variable sets to restrict the lists to the variables you are really >using, but lists with thousands of variables will always be awkward. > >4. Memory usage is not just about the dictionary. The operating >system will almost always be paging code and data between memory and >disk. (You can look at paging rates via the Windows Task >Manager). The more you page, the slower things get, but the variable >dictionary is only one among many objects that the operating system is >juggling. However, there is another effect. On NT and later, Windows >automatically caches files (code or data) in memory so that it can >retrieve it quickly. This cache occupies memory that is otherwise >surplus, so if any application needs it, portions of the cache are >discarded to make room. You can see this effect quite clearly if you >start SPSS or any other large application; then shut it down and start >it again. It will load much more quickly the second time, because it >is retrieving the code modules needed at startup from memory rather >than disk. The Windows cache, unfortunately, will not help data >access very much unless most of the dataset stays >in memory, because the cache will generally hold the most recently >accessed data. If you are reading cases sequentially, the one you >just finished with is the LAST one you will want again. > >5. These points apply mainly to the number of variables. The number >of cases is not subject to the same problems, because the cases are >not generally all mapped into memory by SPSS (although Windows may >cache them). However, there are some procedures that because of their >computational requirements do have to hold the entire dataset in >memory, so those would not scale well up to immense numbers of cases. > >The point of having an essentially unlimited number of variables is >not that you really need to go to that limit. Rather it is to avoid >hitting a limit incrementally. It's like infinity. You never want to >go there, but any value smaller is an arbitrary limit, which SPSS >tries to avoid. It is better not to have a hard stopping rule. > >Modern database practice would be to break up your variables into >cohesive subsets and combine these with join (MATCH FILES in SPSS) >operations when you need variables from more than one subset. SPSS is >not a relational database, but working this way will be much more >efficient and practical with very large numbers of variables. > > >Regards, >Jon Peck >SPSS R & D


Back to: Top of message | Previous page | Main SPSSX-L page