Date: Wed, 28 Nov 2007 12:28:36 -0700
Reply-To: Brian Moore <BMoore@worldatwork.org>
Sender: "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From: Brian Moore <BMoore@worldatwork.org>
Subject: Re: set cache reasonable values
In-Reply-To: A<a78982de0711270942p482f19eev412d8054c0b3b4a6@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Thanks to everyone for insights on this issue.
I have succeeded in cleaning out the cache, but still getting warnings.
Apparently this is the maximum workspace (& from the warning <end of
message> appears to be a software determined maximum, not anything to do
with computing power) and I'm still getting the error.
Other specs that may matter:
-using version 15 (but are looking at upgrading to 16)
-File size is 420 MB
-& I have ~20 gigs total free.
Any other ideas?
Thanks,
Brian
extracts from the output file appear below.
>The parameter of the WORKSPACE subcommand is in terms of kilobytes
(KB).
>It must be at least 6148 and not greater than 2097151.
set workspace 2097151.
show workspace.
System Settings
Keyword Description Setting
WORKSPACE Special workspace memory limit in kilobytes 2097151
>Warning # 44
>The operating system could not allocate a memory segment of the size
>requested by SPSS. Therefore, SPSS can only use the largest block
>available in memory already allocated. Use SHOW WORKSPACE to determine
the
>size of the request and SET WORKSPACE to change it.
-----Original Message-----
From: SPSSX(r) Discussion [mailto:SPSSX-L@LISTSERV.UGA.EDU] On Behalf Of
Gregory Hildebrandt
Sent: Tuesday, November 27, 2007 10:42 AM
To: SPSSX-L@LISTSERV.UGA.EDU
Subject: Re: set cache reasonable values
Richard,
I have talked to the SPSS Help Desk about this issue. Also, I deal with
datafiles approximately the size you are describing.
Here are some of the suggestion, which I've passed on before.
1. Set Workspace (Total RAM memory) = ,say, 300M for Windows. As I
have 2GB of RAM, and a 93GB hard drive on a laptop, with about 22GB
free, I Set Workpace = 300000 (the units are thousands). However, there
havef been times when I apparently Set Workspace too large. If one
employs the command Show Workspace., one can find out what I assume is
the default value.
Another individual at the SPSS Help desk suggested that on 30M might be
a more appropriate.size for Set Workspace.
2. Under "File," select "Cache Data" before running the procedure.
3. Go to Edit >> Options, and determine the location of the temporary
directory. Delete the SPSS files in this directory. In fact, it has
been suggested that I create a new directory for the temporary files,
that it is dedicated to SPSS. This step can be quite important if one
hasn't cleared the temporary files recently..
4. If you are running a Sort command, first do Transform >> Automatic
Recode for the String variables, and run the procedure using the Recoded
variables. Numeric variables are easier to sort.
After doing all this, if I still have problems I reboot the computer.
Frequently, this seems to work best. There must be some way of emptying
out the entire cache on the computer (the temporary storage I only dimly
understand), but no one has ever been able to tell me how to do this.
Greg
On 11/26/07, Richard Ristow <wrristow@mindspring.com> wrote:
>
> At 10:36 AM 11/26/2007, Brian Moore wrote:
>
> >I *think* I need to increase my cache, but wasn't able to find
> >anything in the documentation about the default value or what a good
> >range might be to increase to or toward.
> >
> >My file is roughly 1 million cases & 180 variables, and the former
> >will be increasing over time.
>
> Here's the problem you cite. I'm quoting the error messages in full:
>
> > 222 COMPUTE InDupGrp = MatchSequence > 0.
> > 223 SORT CASES InDupGrp(D).
> >
> >>Error. Command name: SORT CASES
> >>File write error: file name
> >>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
> >>device (DATA1002) This command not executed.
> >
> >>Error # 5817. Command name: SORT CASES The SPSS file sort has
> >>failed. The file remains unsorted. The specific problem is printed
> >>below.
> >
> >>Error # 5822. Command name: SORT CASES I/O error writing to the
> >>sort scratch file.
>
> As Adrian Barnett, said, on the face of it, this is simple: your
> working disk filled, during the sort. That means you need more free
> disk space, not a change in any SPSS setting. The usual estimate is
> you should have free space 2-3 times the file size, to run a sort
well.
>
> HOWEVER, in your case there are probably ways around, besides adding
> more disk space.
>
> In the first place, the code you posted following this sort command,
> doesn't rely on the data being sorted by 'InDupGrp':
>
> > 224 MATCH FILES /FILE = *
> > /DROP = PrimaryLast InDupGrp MatchSequence.
> > 225 VARIABLE LABELS
> > PFirstOrderDate
> > 'Indicator of each first matching case as Primary' .
> > 226 VALUE LABELS PFirstOrderDate
> > 0 'Duplicate Case'
> > 1 'Primary Case'.
> > 227 VARIABLE LEVEL PFirstOrderDate (ORDINAL).
> > 228 FREQUENCIES VARIABLES = PFirstOrderDate .
>
> I suppose you do need the sort later for something, though.
>
> Second, if you do want to sort by a binary variable, it's probably
> faster to split the file and merge. It should take storage for ONE
> copy of your file, and should be reasonably fast. Like this, assuming
> that InDupYes and InDupNo are file names or file handles (NOT dataset
> names) that can be used for the scratch files. Not tested:
>
> DO IF InDupGrp EQ 1.
> . XSAVE OUTFILE=InDupYes.
> ELSE.
> . XSAVE OUTFILE=InDupNo.
> END IF.
> EXECUTE.
>
> ADD FILES
> /FILE=InDupYes
> /FILE=InDupNo.
>
> By the way, I see that earlier in the job, command
>
> > 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).
>
> ran successfully, on the same file. It may have been happenstance,
> with the scratch files just making it the first time; but it's more
> likely that, if the file was already sorted by at least 'cust', SPSS
> used an adaptive-sort algorithm that took advantage of the file's
> already being nearly in order.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except
> the command. To leave the list, send the command SIGNOFF SPSSX-L For a
> list of commands to manage subscriptions, send the command INFO
> REFCARD
>
=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD
=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@LISTSERV.UGA.EDU (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
