Date:         Thu, 1 Feb 2007 23:15:55 -0500
Reply-To:     Don Henderson <donaldjhenderson@HOTMAIL.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
Comments:     RFC822 error: <W> MESSAGE-ID field duplicated. Last occurrence
              was retained.
From:         Don Henderson <donaldjhenderson@HOTMAIL.COM>
Subject:      Re: disk full problem, additional info
In-Reply-To:  <000001c74678$b2023f20$b3c9678e@soeh.ad>
Content-Type: text/plain; charset="us-ascii"

Do you have 200GB free or 200GB?

Do a PROC CONTENTS on all the data sets in that library, e.g., something
like

proc contents data=msp._all_;
run;

and then do the math using either the number of observations times the
observation length, or the data set page size times the number of pages.

Then you also need to factor in that the step where you are concatenating
the data sets needs room for two copies of the data that you are adding more
observations to.

And looking at your log, it appears that your program could be greatly
optimized by using PROC APPEND to add the observations to your base data
set, i.e, instead of:

data msp.msp_net;set msp.msp_net msp.msp_net_9697 ;run;

use something like:

proc append base = msp.msp_net data = msp.msp_net_9697;
run;

This will run much faster since you are not rewriting the large base data
each time and you also won't need to have room for two copies of the data
set.

HTH,
-don h

> -----Original Message-----
> From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of
> Cornel Lencar
> Sent: Thursday, February 01, 2007 10:18 PM
> To: SAS-L@LISTSERV.UGA.EDU
> Subject: Re: disk full problem, additional info
>
> No, no
>
> My msp library has about 200 GB space (on one hard drive)
> The other hard drive, where temporary files are processed, has 23GB
> available memory. I also increased the virtual memory as well (3576 MB
> with
> a max of ~ 6400 MB.
>
> Has the problem something to do with the operating system? I have no clue
> on
> what is happening (given the fact that I looked at CPU usage, temporary
> file
> usage).
>
> Any help is more than welcomed.
>
> Cheers,
>
> Cornel
>
> -----Original Message-----
> From: toby dunn [mailto:tobydunn@hotmail.com]
> Sent: Thursday, February 01, 2007 6:18 PM
> To: clencar@interchange.ubc.ca; SAS-L@LISTSERV.UGA.EDU
> Subject: RE: disk full problem, additional info
>
> Your MSP library is full, you will need to find a bigger hunk disk space
> to
> store your data.
>
>
>
> Toby Dunn
>
> To sensible men, every day is a day of reckoning. ~John W. Gardner
>
> The important thing is this: To be able at any moment to sacrifice that
> which we are for what we could become. ~Charles DuBois
>
> Don't get your knickers in a knot. Nothing is solved and it just makes you
> walk funny. ~Kathryn Carpenter
>
>
>
>
>
>
> From: Cornel Lencar <clencar@INTERCHANGE.UBC.CA>
> Reply-To: Cornel Lencar <clencar@INTERCHANGE.UBC.CA>
> To: SAS-L@LISTSERV.UGA.EDU
> Subject: disk full problem, additional info
> Date: Thu, 1 Feb 2007 20:47:28 -0500
>
> Hi,
>
> I was asking for some help a bit earlier regarding a problem
> with "insuficient resources".
>
> Here is the log:
>
> 7    data msp.msp_net;set msp.msp_net msp.msp_net_9697 ;run;
>
> NOTE: There were 64429884 observations read from the data set MSP.MSP_NET.
> NOTE: There were 18455399 observations read from the data set
> MSP.MSP_NET_9697.
> NOTE: The data set MSP.MSP_NET has 82885283 observations and 8 variables.
> NOTE: DATA statement used (Total process time):
>        real time           7:20.61
>        cpu time            1:14.89
>
>
> 8    data msp.msp_net;set msp.msp_net msp.msp_net1;run;
>
> ERROR: Write to MSP.MSP_NET.DATA failed. File is full and may be damaged.
> NOTE: The SAS System stopped processing this step because of errors.
> NOTE: There were 82885283 observations read from the data set MSP.MSP_NET.
> NOTE: There were 5195064 observations read from the data set MSP.MSP_NET1.
> WARNING: The data set MSP.MSP_NET may be incomplete.  When this step was
> stopped there were
>           88080347 observations and 8 variables.
> WARNING: Data set MSP.MSP_NET was not replaced because this step was
> stopped.
> NOTE: DATA statement used (Total process time):
>        real time           8:41.35
>        cpu time            1:29.89
>
>
> cheers,
>
> cornel
>
> _________________________________________________________________
> Invite your Hotmail contacts to join your friends list with Windows Live
> Spaces
> http://clk.atdmt.com/MSN/go/msnnkwsp0070000001msn/direct/01/?href=http://s
> pa
> ces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-
> us