Jonathan Fry <jon@spss.com> wrote:

[my original post deleted] >-- >Paradoxically, this sort will probably run a lot faster with _less_ memory.

>The sort requires 16M of working memory to sort a 2.4M file because the data are >compressed on disk, but not in memory. I'm sure that file would occupy more than >16M if saved uncompressed.

The file is a plain ASCII raw data file of 2.4meg - not a compressed system file so it doesn't 'grow'.

>It looks like you have the "Special Workspace Memory Limit" in Edit->Options set >to 14000 (K) or thereabouts. Sort Cases will use that much memory space. Even >on a 32M machine, I would guess that, running NT, the machine will be doing extra >paging because the sort is using so much memory.

Yes, that's right - I'd left it set that high after another program I regularly run said it wanted that much.

>If you specify a smaller amount of memory, SORT CASES will use a disk file for >more of the sorting load. Since it will compress the data going to disk, your >machine may well read and write fewer bytes to and from the disk drive than it >would doing the paging required by a larger workspace.

>Even if you were sorting a file small enough to fit in the 14.3M workspace, you >could probably sort it faster with a smaller workspace, since the disk activity >for the sort work file with a small workspace would be less than the disk >activity for paging with a large workspace.

>Try setting the special workspace memory to, say, 4 megabytes. See how long the >sort takes. I'm sure the readers of this group would be interested in your >findings. >-------------------- >Jonathan Fry >SPSS Inc. >jon@spss.com

OK. Here are the results doing it both ways.

sort cases LAWID,LAWPART(a).

(1) With workspace limit set to 14 meg:

The file to be sorted contains 14,183 cases of 1,664 bytes each. 14,614,592 bytes of memory are available to the sort. 18,008 bytes is the minimum in which the sort will run. 26,730,568 bytes would suffice for an in-memory sort.

Successful completion of the sort.

Preceding task required 230.65 seconds elapsed.

(2) With the workspace set to 4 meg: NOTE : I exited from SPSS bewteen runs

-> sort cases LAWID,LAWPART(a).

The file to be sorted contains 14,183 cases of 1,664 bytes each. 4,030,528 bytes of memory are available to the sort. 18,008 bytes is the minimum in which the sort will run. 26,730,568 bytes would suffice for an in-memory sort.

Successful completion of the sort.

Preceding task required 126.24 seconds elapsed.

Certainly the run time is improved, although it still thrashed the disk a lot and took quite a bit longer than I expected. The difference between the time I reported before and these times is because I ran the whole program for these tests, not just the SORT CASES component as I did for the previous msg. There are a lot of DO IF and COMPUTE operations involving string-manipulations.

I am puzzled by the report it gives of the memory it wants. Both runs say they want 26.7 meg for an in-memory sort, although they run in different times. It was this type of report which led me in the first place to increase the workspace limit. Clearly I'm not understanding the import of these messages. Any help in this direction would be greatly appreciated.

Thanks for your input so far

Regards

Adrian

-- Adrian Barnett | "It's always the trombone player" Ph :+61-8-207-1531 | (Faye Dunaway in 'The Arrangement') Fax: +61-8-207-1730 Email: adrianb@dove.mtx.net.au