LISTSERV at the University of Georgia
Menubar Imagemap
Home Browse Manage Request Manuals Register
Previous messageNext messagePrevious in topicNext in topicPrevious by same authorNext by same authorPrevious page (June 2008, week 2)Back to main SAS-L pageJoin or leave SAS-L (or change settings)ReplyPost a new messageSearchProportional fontNon-proportional font
Date:         Fri, 13 Jun 2008 03:06:44 +0530
Reply-To:     ajay ohri <ohri2007@GMAIL.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         ajay ohri <ohri2007@GMAIL.COM>
Subject:      Re: Effective ways of Programming in SAS when dealing with large
              database tables
Comments: To: SUBSCRIBE SAS-L Chandra Gadde <ddraj2015@gmail.com>
In-Reply-To:  <200806122129.m5CJhil3031728@malibu.cc.uga.edu>
Content-Type: text/plain; charset=ISO-8859-1

Now that sounds a fun project !!! Let us know hope the join and the paper turns out.

I have handled at most a 150 gb dataset for merging and the lesson was to first test the code on a 1-5 % random sample (learnt after 1 week :( ),

and if possible to split one of the datasets (in 20 parts) and do the joins in series rather than parallel...computing resources dependent. (learnt after 2 weeks :)) )

using compression techniques also helps before the joins. and sql *seemed*faster than data step (but thats just me...)

HTH

Regards,

Ajay

www.decisionstats.com

On Fri, Jun 13, 2008 at 2:59 AM, SUBSCRIBE SAS-L Chandra Gadde < ddraj2015@gmail.com> wrote:

> 1 TERRABYTE SIZE table. I am working on a project that involves joining two > tabels (one is 2.5 terrabytes and another one is 1 terrabyte). :) > > > On Thu, 12 Jun 2008 14:40:24 -0400, Richard Wright > <richard.wright@DARS.STATE.TX.US> wrote: > > >On Thu, 12 Jun 2008 12:10:25 -0400, SUBSCRIBE SAS-L Chandra Gadde > ><ddraj2015@GMAIL.COM> wrote: > > > >>All > >> > >>I am planning to write a paper in our company on using SAS when dealing > >>with large database tables. Could you please send me some > >>information/websites/papers on this topic. > >> > >>Thanks. > > > >I'm curious - what do you consider large? >


Back to: Top of message | Previous page | Main SAS-L page