LISTSERV at the University of Georgia
Menubar Imagemap
Home Browse Manage Request Manuals Register
Previous messageNext messagePrevious in topicNext in topicPrevious by same authorNext by same authorPrevious page (May 2001, week 5)Back to main SAS-L pageJoin or leave SAS-L (or change settings)ReplyPost a new messageSearchProportional fontNon-proportional font
Date:         Thu, 31 May 2001 12:15:54 +0100
Reply-To:     Peter Crawford <peter.crawford@DB.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         Peter Crawford <peter.crawford@DB.COM>
Subject:      Re: load balancing for parallel processing
Comments: To:
Content-type: text/plain; charset=iso-8859-1

Thank you Paula The atricle is one I had already retrieved from the SI web site, but the underlying objectives (big simulation) and platform context (SMP) are a bit different from mine.

My process transfers approx. 400 sas tables in 50 sets of 8 from os390 to unix/aix (as .csv files). It should run monthly. Set size ranges from 2gigabytes to 600kbytes. The total size of all sets is approx 20gigabytes for April data. Account "churn" causes growth over the year - expected to rise to 30gb. Each set of 8 is an integral unit, within which some restructuring is performed.

My first simple algorithm placed the largest set in its own stream (referred to as 0 ). Then, in descending order of size, the remainder are allocated in order to streams 1 to 9.

That puts the 2nd, 11th, 20th, 29th etc largest in stream1. Within a stream, these execute serially.

stream sets (numbered by descending size) 0 1 1 2 11 20 2 3 12 21 3 4 13 22 4 5 14 23 5 6 15 24 6 7 16 25 7 8 17 26 8 9 18 27 9 10 19 28 etc

Unfortunately, this gives stream 1 more work than stream 2 etc For a first order of improvement, I simply reversed the allocation of sets 11 to 19. I am hoping to discover a method which receives the size of each set, and proposes the efficient combinations which will result in the earliest completion of all streams. The optimisation can complete early if streams 1 to 9 can complete before stream 0.

Many thanks for your interest

Peter Crawford

Datum: 31/05/2001 03:05 An: Peter Crawford/Zentrale/DeuBaExt@Zentrale

Betreff: Re: load balancing for parallel processing Nachrichtentext:

I am not sure exactly what you are up to, but wish the attached paper may be relevant to you. Good Luck.

Paula D ----- Original Message ----- From: "Peter Crawford" <peter.crawford@DB.COM> Newsgroups: Sent: Wednesday, May 30, 2001 1:28 PM Subject: load balancing for parallel processing

> Hi there all you sas programming capacity planners and wizzards > I need your algorithmns ! > 50 jobs of varying sizes are to be executed on the multi-task op system > (this is os390, but the algorithm is probably platform independant). > When all run in just one initiator - one after the other - they take a > few hours (5 - 6 ). > With 10 initiators (not allowed to use more) I want to find a way to > balance the load on each initiator. > The jobs all perform the same process and can be scaled adequately by > the size of its largest table/sas data set. > But when I (weakly) balance the load across 10 initiators, the total > delivery completes in the time for the largest plus 10-15 mins. > That means now 50-60 mins all jobs. The improvement in overall time > for completion is welcome, but I hope to find a way to make the > selection of which jobs should run after each other more automatic. > This is not a new problem (except for me). > I expect there is an algorithm out there - > even a learned List (like sas-l) somewhere. > > If you are able to advise, I would appreciate your help > Regards > Peter Crawford > >

<attachment removed> The %Distribute System for Large-Scale Parallel Computation in the SAS ® System Cheryl Doninger, Manager of SAS/CONNECT ® Software R & D Randy Tobias, Manager of Linear Models R & D


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Back to: Top of message | Previous page | Main SAS-L page