Date:         Mon, 10 May 1999 09:57:42 -0400
Reply-To:     "Kowalczyk, Andrew" <AKowalczyk@NT.DMA.STATE.MA.US>
Sender:       "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From:         "Kowalczyk, Andrew" <AKowalczyk@NT.DMA.STATE.MA.US>
Subject:      Re: Creating Large Data Sets On Tape
Comments: To: Bob Fitz <PW098@AOL.COM>
Content-Type: text/plain; charset="iso-8859-1"

It seems like you have two decisions to make at the start - DASD vs Tape
and  many members in a single library vs separate library for each month.
 
DASD - does your data center policies and budget allow you to put datasets
this big on DASD.  If you have a single library it will eventually get very
large and you will run into issues with spanning packs and the inevitable
call from the capacity planning people.
 
If you put each month in its own library then you get quick access.  If you
expect to mostly work with the recent months and only occasionally with
older data then this makes a lot of sense. Most MVS shops these days have
some form of automatic archiving - so the most recent month will probably
stay "online" since you are using it - the older months will automatically
get saved to tape.  When you need it again the tape will be mounted and that
file loaded back onto DASD.
 
TAPE If you go to tape initially, my advice would definitely be separate
libraries.  This much data will surely be many tapes of data - if you have
many members in a single tape library - and you want to get to the last
member - SAS will read through all the intervening members to get there - a
lot of extra tape mounts.
 
In our shop most datasets this size would be created on tape.  Since we have
a robotic "juke box" to mount the tapes - the time delay to get a tape
mounted is very minimal. (in my mind the biggest draw back to tapes is
waiting for the operator to come back from coffee and mount your tape).
 
If you expect to regularly analyze all 24 months of data in a consolidated
manner - then you really need to think this through - if you expect to say
something like:
SET mon01.f mon02.f mon03.f mon04.f mon05.f ... mon23.f mon24.f; /* ellipsis
is not SAS - just my notation*/
then SAS will try to mount all libraries at once - a problem in either case
- you will need 24 tape drives or all the archived datasets will be loaded
to DASD(assuming your data senter has enough free DASD for this to happen.)
You cannot "concatenate" separate SAS libraries at the JCL level.
 
If consolidated analysis is important I would suggest that you build a
consolidated file, as well as your monthly file.  Perhaps a Year-To-Date
file - then you would only need 2 or three concurrent mounts.
 
Become familiar with the data retention policies at your installation - in
many places new data sets are automatically discarded at the end of the work
day or after 14 days or some such.  You may need to specify a RETPD on you
JCL.  Depending on local policy you may need to fill out paper work or
adhere to some naming convention to be retained properly (or even to create
the dataset in the first place).  Generation Data Groups (GDG) may be
something useful as well.
 
So MY ideal solution would be:
Each month in its own library on DASD - only recent month is online - older
months have been auto-archived.
A separate(multi-)tape library for each Year or Year-To-Date (single member
in each library).
Also a separate tape backup of each month.  If it is important (and why
would you use it if it wasn't important) then an off-site "vault" copy of
each month and the YTD.
 
 
> -----Original Message-----
> From: Bob Fitz [SMTP:PW098@AOL.COM]
> Sent: Sunday, May 09, 1999 7:36 AM
> Subject:      Creating Large Data Sets On Tape
>
> Hi SAS-Ler's,
>
> I am trying to create a SAS program where I write out a large SAS data set
> monthly.  Its 1200 bytes wide and 6 million observations long.  One data
> set
> is not a problem but I'm not sure how to go about creating it monthly.
> I'll
> eventually end up with 24 monthlies.   Is it best to make a separate
> library
> for each or can it all be in one SAS library?  If I put it in one library,
> how would you dimension the size of it? What are some plusses and minuses
> for
> each solution?
>
>  I finally got a hold of the SAS MVS companion but I did not find the
> answer
> there.  I have  Michael Raithel's book on order but I can't wait a couple
> of
> weeks to start coding.   Any help, examples, hints would be greatly
> appreciated.
>
> Best Regards,
> Bob Fitzgerald