Date: Thu, 13 May 2010 11:47:41 +0930
Reply-To: "Barnett, Adrian (DECS)" <Adrian.Barnett2@SA.GOV.AU>
Sender: "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From: "Barnett, Adrian (DECS)" <Adrian.Barnett2@SA.GOV.AU>
Subject: Re: CPU Specifications used for SPSS
In-Reply-To: <OF217EC3CA.34FE2EB9-ON86257721.00568648-86257721.0056D05C@us.ibm.com>
Content-Type: multipart/alternative;
Hi Jonathon
Here is a summary of the results, with the value of WORKSPAVE set at 6148, 100000, 500000, 1000000.
WORKSPACE
Elapsed time, sort
CPU time, sort
Whole task, CPU time
Whole task, elapsed time
No. sorts
Merge, elapsed time
Merge, CPU time
6148
24.60
35.80
55.28
48.66
8.00
24.00
19.50
100000
24.10
37.20
56.88
50.13
8.00
25.90
19.60
500000
34.90
60.90
71.22
50.76
2.00
15.70
10.30
1000000
28.70
46.30
56.41
43.56
2.00
14.80
10.00
Elapsed overall time seems to get worse with increasing memory (counterintuitive) whilst merge times go down as would be expected from having to merge fewer subsets. It tends to support the advice in the manual that you should just let SPSS work out the best way of doing it! ;-)
It is very puzzling (well, to me anyway) that it should make things worse to have more memory available.
Below are the detailed reports from each run
Here's the report from the listing with WORKSPACE set to 1000000:
sort cases locality_name, street_name,st_type,street_suffix
number_first,number_last,flat_number,LOT_NUMBER.
Row size: 856 bytes
Row count: 932355
Uncompressed file size: 761.1MB
Elapsed time for sort phase: 28.7 seconds
Processor time for sort phase: 46.3 seconds
Total size of files written by sort phase: 252.3MB
Sort #1
Memory budget: 500000KB
Row limit: 1340314
Rows sorted: 339763
Input time: 2.8 seconds
Sort time : 11.9 seconds
Output time: 3.7 seconds
Sort #2
Memory budget: 500000KB
Row limit: 592592
Rows sorted: 592592
Input time: 4.7 seconds
Sort time : 19.0 seconds
Output time: 4.9 seconds
Merge # 1
Sequences merged: 2
Elapsed time: 14.8 seconds
Processor time: 10.0 seconds
Preceding task required 56.41 seconds CPU time; 43.56 seconds elapsed.
Below is the same section after re-setting WORKSPACE to default value of 6148
Row size: 856 bytes
Row count: 932355
Uncompressed file size: 761.1MB
Elapsed time for sort phase: 24.6 seconds
Processor time for sort phase: 35.8 seconds
Total size of files written by sort phase: 252.3MB
Sort #1
Memory budget: 51200KB
Row limit: 60681
Rows sorted: 60681
Input time: 0.7 seconds
Sort time : 1.6 seconds
Output time: 1.1 seconds
Sort #2
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 129980
Input time: 1.0 seconds
Sort time : 3.6 seconds
Output time: 2.0 seconds
Sort #3
Memory budget: 51200KB
Row limit: 134089
Rows sorted: 134089
Input time: 1.1 seconds
Sort time : 3.6 seconds
Output time: 1.3 seconds
Sort #4
Memory budget: 51200KB
Row limit: 136533
Rows sorted: 136533
Input time: 1.1 seconds
Sort time : 3.8 seconds
Output time: 2.2 seconds
Sort #5
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 137248
Input time: 1.1 seconds
Sort time : 3.8 seconds
Output time: 2.1 seconds
Sort #6
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 137248
Input time: 1.0 seconds
Sort time : 4.0 seconds
Output time: 1.3 seconds
Sort #7
Memory budget: 51200KB
Row limit: 137608
Rows sorted: 130862
Input time: 1.0 seconds
Sort time : 3.8 seconds
Output time: 3.0 seconds
Sort #8
Memory budget: 51200KB
Row limit: 136889
Rows sorted: 65714
Input time: 0.5 seconds
Sort time : 1.9 seconds
Output time: 1.3 seconds
Merge # 1
Sequences merged: 8
Elapsed time: 24.0 seconds
Processor time: 19.5 seconds
Preceding task required 55.28 seconds CPU time; 48.66 seconds elapsed.
Here it is set to 500,000
Row size: 856 bytes
Row count: 932355
Uncompressed file size: 761.1MB
Elapsed time for sort phase: 34.9 seconds
Processor time for sort phase: 60.9 seconds
Total size of files written by sort phase: 252.3MB
Sort #1
Memory budget: 250000KB
Row limit: 296296
Rows sorted: 296296
Input time: 2.6 seconds
Sort time : 9.0 seconds
Output time: 2.5 seconds
Sort #2
Memory budget: 250000KB
Row limit: 664935
Rows sorted: 636059
Input time: 5.0 seconds
Sort time : 21.8 seconds
Output time: 5.5 seconds
Merge # 1
Sequences merged: 2
Elapsed time: 15.7 seconds
Processor time: 10.3 seconds
Preceding task required 71.22 seconds CPU time; 50.76 seconds elapsed.
And here it is set to 100000
Row size: 856 bytes
Row count: 932355
Uncompressed file size: 761.1MB
Elapsed time for sort phase: 24.1 seconds
Processor time for sort phase: 37.2 seconds
Total size of files written by sort phase: 252.3MB
Sort #1
Memory budget: 51200KB
Row limit: 60681
Rows sorted: 60681
Input time: 0.7 seconds
Sort time : 1.8 seconds
Output time: 1.9 seconds
Sort #2
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 129980
Input time: 1.1 seconds
Sort time : 3.5 seconds
Output time: 2.7 seconds
Sort #3
Memory budget: 51200KB
Row limit: 134089
Rows sorted: 134089
Input time: 1.1 seconds
Sort time : 3.7 seconds
Output time: 1.2 seconds
Sort #4
Memory budget: 51200KB
Row limit: 136533
Rows sorted: 136533
Input time: 1.1 seconds
Sort time : 4.2 seconds
Output time: 1.2 seconds
Sort #5
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 137248
Input time: 1.1 seconds
Sort time : 4.0 seconds
Output time: 1.4 seconds
Sort #6
Memory budget: 51200KB
Row limit: 137248
Rows sorted: 137248
Input time: 1.1 seconds
Sort time : 4.2 seconds
Output time: 1.1 seconds
Sort #7
Memory budget: 51200KB
Row limit: 137608
Rows sorted: 130862
Input time: 1.1 seconds
Sort time : 4.1 seconds
Output time: 1.3 seconds
Sort #8
Memory budget: 51200KB
Row limit: 136889
Rows sorted: 65714
Input time: 0.6 seconds
Sort time : 2.0 seconds
Output time: 0.8 seconds
Merge # 1
Sequences merged: 8
Elapsed time: 25.9 seconds
Processor time: 19.6 seconds
Preceding task required 56.88 seconds CPU time; 50.13 seconds elapsed.
Adrian Barnett
Project Officer
Educational Measurement and Analysis
Data and Educational Measurement
DECS
ph 82261080
________________________________
From: Jon Fry [mailto:JonathanFry@us.ibm.com]
Sent: Thursday, 13 May 2010 1:18 AM
To: Barnett, Adrian (DECS)
Cc: 'SPSSX-L@LISTSERV.UGA.EDU'
Subject: RE: CPU Specifications used for SPSS
Adrian,
For the two-sort version, what are the reported read, sort, and write times?
Jonathan
[text/html]
