Date: Tue, 10 Nov 1998 15:14:18 -0500
Reply-To: "Stetz, Cynthia (PCM - NJ)" <CStetz@NA2.US.ML.COM>
Sender: "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From: "Stetz, Cynthia (PCM - NJ)" <CStetz@NA2.US.ML.COM>
Subject: Re: Funky Stuff in SAS?
Content-Type: text/plain; charset="iso-8859-1"
Karsten and all,
One way around the finite size of a UNIX filesystem when using the SAS large
file support is to define a library across several physical filesystems (or
directories). The catch here is that SAS will only put one partition of the
large file in each directory, and the partsize is limited to the smallest
available slice of space in any of the directories.
If the original poster of this problem would like to discuss this further,
you may mail me directly at cynthia_stetz@ml.com
<mailto:cynthia_stetz@ml.com>
I'm sorry, but I deleted the original post.
Regards,
Cynthia
-----Original Message-----
From: Self, Karsten [SMTP:Karsten.Self@SCHWAB.COM]
Sent: Tuesday, November 10, 1998 2:50 PM
To: SAS-L@UGA.CC.UGA.EDU
Subject: Re: Funky Stuff in SAS?
I disagree with Bernard's diagnosis. Disk full results in a dialog or
message "out of resources" or "insufficient space". Baba is running
HPUX. SAS does not offer native large file (> 2GB) support on this
platform, though a large file experimental mode is available. SASWORK
could be 10 GB or 10 TB, he still can't produce a single system file of
more than 2 GB.
The workaround is to enable SAS large file support. The confirmation is
to watch file size and disk utilization within SAS WORK.
You typically can't allocate space in Unix as on MVS. A filesystem,
once built, has a finite size, and is modifiable only by the root user.
Users may have quotas or resource limits imposed (man 'quotas' and
'ulimit'), though this usually isn't the case. You *can* point to a
larger space, if one is available, by specifying a different WORK system
option at SAS startup.
2 GB is roughly:
1m records X 2k lrecl
2m records X 1k lrecl
4m records x 500 bytes lrecl
...a pattern should be emerging.
My usual question is whether or not the sort is necessary. It often
isn't, or can be postponed, if you can do subsetting with a format,
apply CLASS variables to a summary, or use an index. A standard
practice is:
sort
subset
summarize
A much more efficient method is:
subset
summarize (based on existing collation sequence)
sort
summarize
...tossing as much of the data as possible along the way and using views
where possible. The NOTSORTED option can be used to allow summarization
of data in which some grouping of like keys exists, but overall ordering
isn't guaranteed.
I responded to an optimization problem last December on SAS-L which
makes an interesting case study. Original problem, tactics, final code,
and optimizations, are available in the following links.
http://x8.dejanews.com/getdoc.xp?AN=309293810
http://www.dejanews.com/getdoc.xp?AN=299366238
Karsten M. Self (Karsten.Self@schwab.com)
Trilogy Consulting
What part of "gestalt" don't you understand?
WARNING: All e-mail sent to or from this address will be received by
the Charles Schwab corporate e-mail system and is subject to archival
and review by someone other than the recipient.
> ----------
> From: Ashiru, Babatunde[SMTP:AshiruB@WHITEOAKSEMI.COM]
> Reply To: Ashiru, Babatunde
> Sent: Tuesday, November 10, 1998 11:09 AM
> To: SAS-L@UGA.CC.UGA.EDU
> Subject: Re: Funky Stuff in SAS?
>
> Ok, points well taken but I have gotten as far as several hundred
> thoudsands
> over 1 million records in the past without any problem...albeit I
> seldom get
> I/O error message. We thought it was a tmp space problem; allocated
> more
> space and/or moved the process to another server with over 10
> gigabytes of
> hard-disk space but all to no avail?
>
> BRA
>
> -----Original Message-----
> From: Bernard Tremblay [SMTP:bernard@capitale.qc.ca]
> Sent: Tuesday, November 10, 1998 2:05 PM
> To: AshiruB@WHITEOAKSEMI.COM
> Subject: Re: Funky Stuff in SAS?
>
> Hi,
>
> This funky stuff is called : DISK FULL !
>
> You're out of space on the volume that is used for
> sorting
> (usualy the work library) or the volume where is the
> 'PRDCNTDS'
> library.
> Try allocate more work space or find another volume for it...
>
> regards,
> Bernard Tremblay
>
