Date: Thu, 31 May 2001 12:15:54 +0100
Reply-To: Peter Crawford <peter.crawford@DB.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Peter Crawford <peter.crawford@DB.COM>
Subject: Re: load balancing for parallel processing
Content-type: text/plain; charset=iso-8859-1
Thank you Paula
The atricle is one I had already retrieved from the SI web site, but
the underlying objectives (big simulation) and platform context (SMP)
are a bit different from mine.
My process transfers approx. 400 sas tables in 50 sets of 8
from os390 to unix/aix (as .csv files). It should run monthly.
Set size ranges from 2gigabytes to 600kbytes. The total size of all
sets is approx 20gigabytes for April data. Account "churn" causes
growth over the year - expected to rise to 30gb. Each set of 8 is an
integral unit, within which some restructuring is performed.
My first simple algorithm placed the largest set in its own stream
(referred to as 0 ). Then, in descending order of size, the remainder
are allocated in order to streams 1 to 9.
That puts the 2nd, 11th, 20th, 29th etc largest in stream1.
Within a stream, these execute serially.
stream sets (numbered by descending size)
1 2 11 20
2 3 12 21
3 4 13 22
4 5 14 23
5 6 15 24
6 7 16 25
7 8 17 26
8 9 18 27
9 10 19 28 etc
Unfortunately, this gives stream 1 more work than stream 2 etc
For a first order of improvement, I simply reversed the allocation of
sets 11 to 19.
I am hoping to discover a method which receives the size of each set,
and proposes the efficient combinations which will result in the
earliest completion of all streams. The optimisation can complete
early if streams 1 to 9 can complete before stream 0.
Many thanks for your interest
Datum: 31/05/2001 03:05
An: Peter Crawford/Zentrale/DeuBaExt@Zentrale
Betreff: Re: load balancing for parallel processing
I am not sure exactly what you are up to, but wish the attached paper may be
relevant to you. Good Luck.
----- Original Message -----
From: "Peter Crawford" <peter.crawford@DB.COM>
Sent: Wednesday, May 30, 2001 1:28 PM
Subject: load balancing for parallel processing
> Hi there all you sas programming capacity planners and wizzards
> I need your algorithmns !
> 50 jobs of varying sizes are to be executed on the multi-task op system
> (this is os390, but the algorithm is probably platform independant).
> When all run in just one initiator - one after the other - they take a
> few hours (5 - 6 ).
> With 10 initiators (not allowed to use more) I want to find a way to
> balance the load on each initiator.
> The jobs all perform the same process and can be scaled adequately by
> the size of its largest table/sas data set.
> But when I (weakly) balance the load across 10 initiators, the total
> delivery completes in the time for the largest plus 10-15 mins.
> That means now 50-60 mins all jobs. The improvement in overall time
> for completion is welcome, but I hope to find a way to make the
> selection of which jobs should run after each other more automatic.
> This is not a new problem (except for me).
> I expect there is an algorithm out there -
> even a learned List (like sas-l) somewhere.
> If you are able to advise, I would appreciate your help
> Peter Crawford
The %Distribute System for Large-Scale Parallel Computation in the SAS ® System
Cheryl Doninger, Manager of SAS/CONNECT ® Software R & D
Randy Tobias, Manager of Linear Models R & D
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.