Date: Mon, 26 Apr 1999 10:50:22 -0400
Reply-To: ABELSOR <ABELSOR@WESTAT.COM>
Sender: "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From: ABELSOR <ABELSOR@WESTAT.COM>
Subject: Re: (MVS) Re: How to trick Proc Freq?
Content-Type: text/plain; charset=US-ASCII
Some people are cumudgeons just on principle:-)
Seriously, I have known managers like Mr. Pider purports to be. What they don't
understand, they automatically distrust. And, since they're very busy people,
they don't take the time to try to understand these things. The result is
Subject: (MVS) Re: How to trick Proc Freq?
Author: RAITHEM <RAITHEM@WESTAT.COM>
Date: 4/26/99 10:07 AM
David Pider posted Part II of his anti-Guru diatribe. (His entire posting can
be found beneath the Sig line, below).
David, what a completely ignorant, non-constructive, and non-informational
posting you have proffered! It is hard to tell whether you are really so angry
and misguided or whether you are one of those people who gets a personal thrill
out of "kicking over an anthill" just to see what will happen. Either way; it
is a sad commentary on where you have come to be. I would tell you that it does
not have a place on SAS-L, but I am sure that it will not do any good.
"Technical Manager." "...just fired..." "15 years of SAS experience" Yea,
Since we will undoubtedly continue to hear from you, I will definitely read your
future diatribe postings and enjoy them for their rich, though perverse,
entertainment value. Though they are not as much fun as some of the off-topic
jokes, they sure do break the routine.
I hope that this suggestion proves helpful now, and in the future!
Of course, all of these opinions and insights are my own, and do not reflect
those of my organization or my associates.
Michael A. Raithel
"The man who wrote the book on performance."
Author: Tuning SAS Applications in the MVS Environment
Your words waste and decay, nothing you say reaches my ears anyway -- Emerson,
Lake and Palmer
Subject: Re: How to trick Proc Freq?
Author: david pider <dpider@HOTMAIL.COM>
Date: 4/26/99 12:31 AM
Alex Martchenko wrote:
> I have a credit policy SAS data set with almost 100 million
>observations. It has a numeric variable FLDR_ID that can be any
>integer number from -500,000 to +500,000. So on the average there are
>about 100 observations with the same FLDR_ID on the file. The data
>set is sorted by a different variable (account number) and so it's
>unsorted by FLDR_ID. I'm trying to create a data set with
>frequencies, cumulative frequencies, percents and cumulative percents
>for all values of FLDR_ID. I'm trying to use proc freq like that:
> proc freq data=folders(keep=fldr_id);
> tables fldr_id / missing out=fldr_fq;
>But I have 2 problems: 1) proc freq runs out of memory and abends
>the job 2) if I run it on a much smaller subset so it runs OK, it
>doesn't create cumulative variables in the OUT= file. I could live
>with problem 2 because I can read the output data set with frequency
>and then calculate cumulatives in a data step.
> Real pain is problem 1. I run it in batch on OS/390, 6.09E. A job
>can only use 50M of memory (that's all system people say we can have
>for a single SAS job). But proc freq wants more, so if I specify
>MEMSIZE=100M it doesn't matter because after 50M limit is exceeded,
>the job abends anyway.
If you read SAS manuals you wouldn't be in trouble. About PROC FREQ,
the manual says 'the maximum number of levels allowed for any one
variable is 32,767. If you have a variable with more than 32,767
levels, use the SUMMARY procedure'.
>I know I can sort it by FLDR_ID and then use first. and last.
>variables to do what I need. But in our system sorting 100 million
>records is a big problem itself. Does anyone have suggestions? I will
>greatly appreciate any input.
Here is one: if some of self appointed 'gurus' offer you a 'datastep
algorithm' claiming it is 'more efficient' that SAS procs do yourself
a favor and use 'delete' button. Trust my 15 years in SAS, none of
thses 'solutions' run faster or 'more efficiently' than properly
applied SAS procs. I've just fired one 'expert' as soon as I saw that
he tried to get smart and summarized in datastep instead of proc
summary. If I can't ban them 'gurus' from SAS profession, at least I
can keep them away from my shop. And it's getting worse. Now this
'datastep approach' have even sneaked into SUGI. I almost fainted at
seeing a paper by one of them 'gurus' (who I bet have no idea of real
world programing) trying to replace proc sort with his 'quick sort'
monstrosity. I wonder why he didn't call it 'blast sort' or something?
BTW, try SQL too. Personally, I don't like it (it shouldn't of been in
SAS in the first place) but at least it's a SAS proc and at least will
beat any 'datastep algorithm' hands down.
15 years of SAS experience
Get Free Email and Do More On The Web. Visit http://www.msn.com