Date: Sat, 5 Feb 2011 19:11:02 -0800
Reply-To: Jack Hamilton <jfh@STANFORDALUMNI.ORG>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Jack Hamilton <jfh@STANFORDALUMNI.ORG>
Subject: Re: checking efficiency by running codes, reliable?
In-Reply-To: <E0B423A8C0D1E74B8905B2C5CB38C1AF128FB4@GENO4.wharton.upenn.edu>
Content-Type: text/plain; charset=us-ascii
Also, even if you turned off SAS caching, the OS might be doing its own caching.
On Feb 5, 2011, at 6:28 PM, Keintz, H. Mark wrote:
> And ... even running the same data step (or proc step) twice in a row, can yield dramatically different results. In my experience this most commonly arises because the second step benefits from disk cacheing initiated by the first step. Of course, this also means that two DIFFERENT techniques run in succession against the same input dataset can result in the second technique getting the benefit of cacheing.
>
> So, if you really want a reliable test with no between-test cacheing effect, you might have to reboot the system between tests. I actually did this on my windows machine when I wanted to compare some indexing techniques on the same dataset. (Simply using the SAS option to preclude cacheing would not serve in my case, because I wanted the benefit of within-test caching).
>
> Regards,
> Mark Keintz
>
>
>> -----Original Message-----
>> From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of
>> Arthur Tabachneck
>> Sent: Saturday, February 05, 2011 6:40 PM
>> To: SAS-L@LISTSERV.UGA.EDU
>> Subject: Re: checking efficiency by running codes, reliable?
>>
>> Max,
>>
>> And, in addition to what Joe said, run a number of replications, and
>> reverse
>> the order or what is run when, and do everything you can do reduce what
>> is
>> actually noise.
>>
>> No, it is definitely not straight forward. And, with 9.2, it appears
>> that
>> there are even more factors that one has to consider then there used to
>> be.
>>
>> Art
>> ---------
>> On Sat, 5 Feb 2011 17:32:53 -0600, Joe Matise <snoopy369@GMAIL.COM>
>> wrote:
>>
>>> You have other variables that affect your CPU/Real times that aren't
>> easily
>>> controlled, but tend to have a small impact on large runtimes. So if
>> you
>>> want to compare efficiency you need to compare using a large dataset
>> that
>>> will take at least minutes to complete.
>>>
>>> -Joe
>>>
>>> On Sat, Feb 5, 2011 at 5:17 PM, bbser 2009 <bbser2009@gmail.com>
>> wrote:
>>>
>>>> We usually compare efficiency of different codes by direct running
>> them.
>>>> Do you think such a method is reliable? I doubt it based on this
>> fact:
>>>> I ran the same code three times and got three very distinct sets of
>> cpu
>> and
>>>> real times.
>>>> What happened? Thanks.
>>>>
>>>> Max
>>>>
>>>> ==================
>>>>
>>>> 2873 proc sql;
>>>> 2874 select
>>>> 2875 destination,
>>>> 2876 average,
>>>> 2877 max,
>>>> 2878 late/(late+early) as Prob
>>>> 2879 from( select
>>>> 2880 destination,
>>>> 2881 avg(delay) as average,
>>>> 2882 max(delay) as max,
>>>> 2883 sum(delay>0) as late,
>>>> 2884 sum(delay<=0) as early
>>>> 2885 from sasuser.flightdelays
>>>> 2886 group by destination)
>>>> 2887 order by average
>>>> 2888 ;
>>>> NOTE: Writing HTML Body file: sashtml147.htm
>>>> 2889 quit;
>>>> NOTE: PROCEDURE SQL used (Total process time):
>>>> real time 0.13 seconds
>>>> cpu time 0.05 seconds
>>>>
>>>>
>>>>
>>>> 2901 proc sql;
>>>> 2902 select
>>>> 2903 destination,
>>>> 2904 average,
>>>> 2905 max,
>>>> 2906 late/(late+early) as Prob
>>>> 2907 from( select
>>>> 2908 destination,
>>>> 2909 avg(delay) as average,
>>>> 2910 max(delay) as max,
>>>> 2911 sum(delay>0) as late,
>>>> 2912 sum(delay<=0) as early
>>>> 2913 from sasuser.flightdelays
>>>> 2914 group by destination)
>>>> 2915 order by average
>>>> 2916 ;
>>>> NOTE: Writing HTML Body file: sashtml149.htm
>>>> 2917 quit;
>>>> NOTE: PROCEDURE SQL used (Total process time):
>>>> real time 0.03 seconds
>>>> cpu time 0.01 seconds
>>>>
>>>>
>>>> 2918 proc sql;
>>>> 2919 select
>>>> 2920 destination,
>>>> 2921 average,
>>>> 2922 max,
>>>> 2923 late/(late+early) as Prob
>>>> 2924 from( select
>>>> 2925 destination,
>>>> 2926 avg(delay) as average,
>>>> 2927 max(delay) as max,
>>>> 2928 sum(delay>0) as late,
>>>> 2929 sum(delay<=0) as early
>>>> 2930 from sasuser.flightdelays
>>>> 2931 group by destination)
>>>> 2932 order by average
>>>> 2933 ;
>>>> NOTE: Writing HTML Body file: sashtml150.htm
>>>> 2934 quit;
>>>> NOTE: PROCEDURE SQL used (Total process time):
>>>> real time 0.04 seconds
>>>> cpu time 0.00 seconds
>>>>
| 
