Date:         Fri, 13 Jun 2008 03:06:44 +0530
Reply-To:     ajay ohri <ohri2007@GMAIL.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         ajay ohri <ohri2007@GMAIL.COM>
Subject:      Re: Effective ways of Programming in SAS when dealing with large
              database tables
Comments: To: SUBSCRIBE SAS-L Chandra Gadde <ddraj2015@gmail.com>
In-Reply-To:  <200806122129.m5CJhil3031728@malibu.cc.uga.edu>
Content-Type: text/plain; charset=ISO-8859-1

Now that sounds a fun project !!! Let us know hope the join and the paper
turns out.

I have handled at most a 150 gb dataset for merging and the lesson was to
first test the code on a 1-5 % random sample  (learnt after 1 week :( ),

and if possible to split one of the datasets (in 20 parts) and do the joins
in series rather than parallel...computing resources dependent. (learnt
after 2 weeks :)) )

using compression techniques also helps before the joins. and sql
*seemed*faster than data step (but thats just me...)

HTH

Regards,

Ajay

www.decisionstats.com


On Fri, Jun 13, 2008 at 2:59 AM, SUBSCRIBE SAS-L Chandra Gadde <
ddraj2015@gmail.com> wrote:

> 1 TERRABYTE SIZE table. I am working on a project that involves joining two
> tabels (one is 2.5 terrabytes and another one is 1 terrabyte). :)
>
>
> On Thu, 12 Jun 2008 14:40:24 -0400, Richard Wright
> <richard.wright@DARS.STATE.TX.US> wrote:
>
> >On Thu, 12 Jun 2008 12:10:25 -0400, SUBSCRIBE SAS-L Chandra Gadde
> ><ddraj2015@GMAIL.COM> wrote:
> >
> >>All
> >>
> >>I am planning to write a paper in our company on using SAS when dealing
> >>with large database tables. Could you please send me some
> >>information/websites/papers on this topic.
> >>
> >>Thanks.
> >
> >I'm curious - what do you consider large?
>