Date: Tue, 6 Apr 2010 10:59:24 -0400
Reply-To: Philip Rack <PhilRack@MINEQUEST.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Philip Rack <PhilRack@MINEQUEST.COM>
Subject: Re: Running SAS batch programs asynchronously
In-Reply-To: <037AB3FF38D44C4BAFB5DFF3D06B57BA40E724A6@EX-CMS01.westat.com>
Content-Type: text/plain; charset="us-ascii"
This is an interesting discussion. About 18 months ago, I wrote a set of
macro's called MPExec that does something similar to what the OP wants.
Basically, it allows MultiProcessing of different parts of a SAS language
program much like SAS/Connect does. I wrote this for WPS and tried to port
it back over to SAS and ran into some issues. Mainly, my problem was
detecting file lock conditions in SAS.
Regarding writing to a data set at the completion of a subprogram, that will
fail under SAS when you are running many subprograms that complete at the
same time. It seems that Windows caches data sets quite well and I ran into
scenario's where the data set was locked and I couldn't write back to it.
The other method (the one I took) was to test if the log files existed. If I
was running say, four subprograms, I would:
1) test if all four logs existed (one for each subprogram)
2) if they existed, then try to open the first one for a read.
2.1) if I couldn't open for a read (file in use) I would wait for a few
seconds and try again.
3.) If I could read the log, I would scan it looking for the text-
"NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414"
or for WPS, "NOTE: Submitted statements took :"
If I could read the text above, I knew that subprogram completed.
4.) Once I could read all four log files, I was certain that the listing
files would also be complete (if they existed)
5). Read all the log files and the listing files back into the WPS/SAS
workspace.
The docs for MPEXEC can be found at:
http://www.minequest.com/Misc/mpexec_users_guide.pdf
or tinyurl: http://tinyurl.com/yh4utas
Phil
Philip Rack
MineQuest, LLC
SAS & WPS Consulting and WPS Reseller
Tel: (614) 457-3714
Web: www.MineQuest.com
Blog: www.MineQuest.com/WordPress
-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Michael
Raithel
Sent: Tuesday, April 06, 2010 9:31 AM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Running SAS batch programs asynchronously
Dear SAS-L-ers,
Scott Bass further clarified his original interesting post with the
following:
> Hi Michael,
>
> Thanks a lot for your reply. I re-read my post, and realize I was too,
> ahem, "economical" in my environment description:
>
> > Environment: Windows XP, SAS 9.1.3 SP 4, using PC-SAS (DMS/batch),
> > SAS/Connect, LSF is not licensed.
>
> Should be:
>
> Environment: Windows XP, SAS 9.1.3 SP 4, using PC-SAS (DMS/batch),
> >>> SAS/Connect and LSF are not licensed. <<<
>
> So sorry for not being clearer in my description. Your posts are
> always
> good stuff, so I will certainly read the links you included.
>
Scott, no worries, from time-to-time we all find that one of our postings
was a bit "too economical":-)
So, I was about to suggest that you create a "checkpoint file", but I see
that SAS-L smarty Jim Groeneveld beat me to it. I like his idea, except
that I would do it a bit differently. I would:
1. Create three, four, or more SAS shell programs; one for each group: G1,
G2, G3, G4 (No political overtones are meant by the latter). Each shell
program consists of four %INCLUDES; one for each program in the group:
%INCLUDE P1, %INCLUDE P2, etc. <--obviously conceptual, not syntactically
correct!
2. Create a simple, checkpoint SAS data set that only contained a couple of
variables, say: jobid, datetime, completion code. Each program that
executes (no matter what group it is in) will write to the checkpoint SAS
data set as the next-to-last thing it does.
3. Create a SAS program that is %INCLUDE-d in each of the GROUP1 jobs--as
the very last thing in each job. That program reads the checkpoint SAS data
set to see if the other programs in the group (P1, P2, P3, P4,<-minus
itself, of course) have successfully completed. If so, it %INCLUDES the
shell program for the next group: %INCLUDE G2, %INCLUDE G3, etc. If not,
it simply ceases to execute. So, the last program asynchronously executing
in a given group will fire off the next group with its last, gasping, dying
breath. How noble!
4. Either make the program in #3 general enough so it can be reused in the
other groups (G2, G3, etc.), or create a special one for the other groups.
5. Use the Windows scheduler to fire off the first SAS shell program that
%INCLUDEs and executes Group 1.
The only obvious problem area is getting the completion code information
into the checkpoint SAS data set. SAS programs that complete successfully
should have no trouble writing to the checkpoint SAS data set. But, those
that error-out present a challenge.
Scott, well, that is my BIG IDEA for the day. Best of luck in all your SAS
endeavors!
I hope that this suggestion proves helpful now, and in the future!
Of course, all of these opinions and insights are my own, and do not reflect
those of my organization or my associates. All SAS code and/or methodologies
specified in this posting are for illustrative purposes only and no warranty
is stated or implied as to their accuracy or applicability. People deciding
to use information in this posting do so at their own risk.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Michael A. Raithel
"The man who wrote the book on performance"
E-mail: MichaelRaithel@westat.com
Author: Tuning SAS Applications in the MVS Environment
Author: Tuning SAS Applications in the OS/390 and z/OS Environments, Second
Edition
http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=58172
Author: The Complete Guide to SAS Indexes
http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=60409
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When I die, I'm leaving my body to science fiction. - Steven Wright
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
