LAG2/etc. are not relevant for this discussion (except as an example of the other weird things that may happen in SAS). LAG/LAG2/etc. do not operate on the previous records at all, but create a new stack (separate from the location of the read-in set data).

-Joe

On Fri, May 20, 2011 at 12:36 PM, Gerhard Hellriegel < gerhard.hellriegel@t-online.de> wrote:

> I think there must be a buffer for the last "few" records. Physical that > is right for sure: the input buffer contains much more than only one > record. > Let's have a kind of logical view on it: > > for the mentioned first / last logic, what's needed? > first FIRST: if a new record is filled in PDV, the BY variable has to be > compared to the previous content. To do that, the previous record must be > anywhere (ok, not the whole record, in our case only the by list, but it > seems to be easier to store the whole record. I don't think, that the > records are moved through memory, but only a pointer change is necessary > to get it. Means, the current pointer contains the start address of the > actual (new) record and the previous pointer is still present.) > Comparing the by-variables sets the first.variable(s) to 1 or 0. > When the next record is read, the "old" pointer and it's memory can be > reused for the next content, it is not needed any more. (in truth it IS > needed and cannot be overwritten - think of LAG2! And how Mike pointed > out, there are much more in a buffer and each obs gets his own pointer) > Do we need also a memory place for the "next obs"? I don't think so for > FIRST. > Now LAST: that cannot be accomplished by comparing with the previous > content. The information must come from comparing to the next record. So > for that, the following record must be present while the current is > processed. > > So we need at least 3 records in memory, the previous, the current and the > next. For other purpose, e.g. for the LAG2 logic, some more. > > That influences also the buffer management: the idea of buffers is to > avoid slow sequential reading and writing. Reading and writing 1000 > records in one block is much faster than reading / writing 1000 times only > one. Also it is possible to work with a buffer at the same time when > another is read or written. It is not necessary to wait until the IO > operation is finished. > What might be a problem is: what do you do when a buffer is at his end, > means, you reach the last record. If you now switch to another buffer and > the old one is overwritten with another (new) 1000 records, the previous > record is gone as soon as the buffer is switched. So the refilling of a no- > longer-needed buffer is not a easy thing. There must be always a small > amount of memory for each buffer to contain some records at the start- > point and for the end. > > Another thing is, the management must always "know" which obs are > currently in memory. If you use direct access with point=, the management > must decide if the needed obs is already in buffer, or is still on disk, > which causes a refilling of a buffer. > In that case it is not sure what's better: very big buffers to get a > better chance of hits, or smaller buffer to reduce IO time. > For "normal" sequential access, which in 99.9% of all SAS programs is > used, some rather big buffers are good. Also if direct access is needed > (think SQL might need such things) and the tables are not too big. You > have the chance to get whole datasets in buffer. > > By the way: on mainframes you should use "half-track" buffers which is > related to the IO routines on the (in our days virtual) DASD. But I don't > think you talk about zOS - SAS... > > Gerhard > > > > > > > On Fri, 20 May 2011 12:52:52 +0000, Michael Raithel > <michaelraithel@WESTAT.COM> wrote: > > >Dear SAS-L-ers, > > > >Haikuo posted the following: > > > >> Dear SAS_Lers, > >> > >> I am sorry if this question has been asked before. > >> > >> I have been taught many times that SAS deals with data on "one record" > >> basis. This can be seen in the mechanism of PDV among others. What > >> puzzles me is that SAS behavior seems changed after "by" statement. In > >> short, how could SAS determine the value of "FIRST.VARIBLE" or > >> "LAST.VARIBLE" without foreseeing the next record? > >> > >Haikuo, I think that only the brainiacs at the SAS Institute can answer > your question properly, but the true answer likely lies in the way that > SAS reads SAS data sets into memory. > > > >The BUFNO= option specifies the number of buffers that SAS allocates for > each data set it opens (CBUFNO= is the related option for SAS catalogs). > In a SAS data set, observations are stored on data set "pages", which are > actually data "blocks" on your storage media. Each of the buffers that is > allocated in memory via the CBUFNO= option is the size of the page of the > related data set. (See the BUFSIZE= and CBUFSIZE= options for SAS data > set and SAS catalog page size, respectively). > > > >When SAS reads a SAS data set, it reads pages from the media and > transfers them into the buffers in memory. So, SAS has scores, dozens, > hundreds, thousands, etc.--depending upon the page size and the > observation size--of observations in memory after the transfer. > Thereafter, individual observations are read into/built in the area of > memory known as the PDV. Nonetheless, SAS has access to all of the > observations that are currently in the buffers in memory. So, that is > likely where all of the action is taking place with SAS doing the FIRST. > and LAST. processing. > > > >This is a fascinating question, and I would love for a Birdie with some > insight to set the record straight. Hint... hint... > > > >Haikuo, best of luck in all your SAS endeavors! > > > >I hope that this suggestion proves helpful now, and in the future! > > > >Of course, all of these opinions and insights are my own, and do not > reflect those of my organization or my associates. All SAS code and/or > methodologies specified in this posting are for illustrative purposes only > and no warranty is stated or implied as to their accuracy or > applicability. People deciding to use information in this posting do so at > their own risk. > > > >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >Michael A. Raithel > >"The man who wrote the book on performance" > >E-mail: MichaelRaithel@westat.com > > > >Author: Tuning SAS Applications in the MVS Environment > > > >Author: Tuning SAS Applications in the OS/390 and z/OS Environments, > Second Edition > >http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=58172 > > > >Author: The Complete Guide to SAS Indexes > >http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=60409 > > > >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >SAS buffers are the best performance tools in recent memory. - Michael A. > Raithel > >+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >