2002 Performance / Price Sort and PennySort
Peng Liu, Yao Shi, Li Zhang, Kuo Zhang, Tian Wang, ZunChong Tian, and Hao Wang. (Coach: Xiaoge Wang)
at
High Performance Institute, Department of Computer Science &
Technology,
Tsinghua University,
Last updated: Dec. 25, 2002 You are the No.
visitor to our site since
News: We have won! News: The benchmark is published The Report
Results Certification Download the Report &
THSort About the Main Authors About the Award Links
Jim Gray Awarded Tsinghua
Team
2002 PennySort (Daytona)
Homepage: http://hpclab.cs.tsinghua.edu.cn/~liupeng (in English)
Peng Liu’s Homepage:
http://hpclab.cs.tsinghua.edu.cn/~pengliu/ (in Chinese)
2002 PennySort Team from Tsinghua University with
Jim Gray
2002 Performance / Price Sort and PennySort
Peng Liu, Yao Shi,
Li Zhang Coach: Xiaoge
Wang
pengliu@ieee.org, shiyao00@mails.tsinghua.edu.cn,
tension7@163.com, wangxg@ tsinghua.edu.cn
High
Performance Institute, Dept. of Computer Science & Tech., Tsinghua
University, Beijing 100084, China
Abstract: Sort
Benchmark[1] is a set of world’s
records that evaluate the progress that computer technology has been making on
transaction processing. In the various sort benchmarks, the PennySort and
Performance / Price Sort were defined to test the maximum cost efficiency of
sort machines. Our program, THSort, is a two passes
external sort program designed to exploit the potential of inexpensive general
machines. Runs on our customized computer, THSort is
able to sort 9.8 GB data[2] (105,000,000
records of 100 bytes each) for a penny, quite double last year’s record (4.19 GB). The paper
presents our considerations when we custom our system, and reports its
PennySort and Performance / Price Sort results, as well as Datamation
Sort and Minute Sort results. The paper also addresses the necessity that
PennySort to be revised to Performance / Price Sort, and provides a simpler
method to calculate Performance / Price Sort result.
About the team: High Performance Institute at
Tsinghua University is the first one that started research on clusters (since 1995)
and grid (since 1998) in China. We focus on parallel/distributed high
performance computing and high performance server. That is why we get
fascinated in the sort benchmark, since we are curious about how to release the
potential I/O capability of the computer. Moreover, as sort is a most common
and time consuming task in transaction processing, sort benchmark can be used
as an important performance criteria of the servers, especially the database
servers. Our sort benchmark team also includes Kuo
Zhang, Tian Wang, ZunChong Tian, and Hao Wang. In the past
few weeks, we have divided our team into two groups, working in collaboration
and competition. This report also includes their contributions.
2002 Daytona PennySort
Hardware Choices: Since the price of hardware drops
sharply and newer components come out one by one, we consider choose hardware
that has new feathers other than the previous winner’s choices.
Motherboard is the No.1 thing to
consider. Once we decided what motherboard to use, we are bound to different
type of CPU, memory, and method of using RAID. Having read many benchmark
reports, we chose Abit KR7A-RAID motherboard, which
use VIA KT266A chipset that support DDR 266 and AMD Athlon
CPU. Another advantage came with KR7A-RAID is that it has an on-board RAID
controller, HPT372 that support RAID 0 and 1, which saved our cost by omitting
a RAID controller card. Actually, motherboards that support KT333 also came out
these days, but none of them has integrated RAID controller yet, and DDR 333 memory
brings higher cost while yield little improvements due to the immaturity of
KT333 chipset.
As most previous winners pointed
out that sort times were very much I/O bound[3], we built two RAID 0 disk arrays. Each
consists two 40GB IBM 120GXP disk drives. Though http://www.storagereview.com tells us
that 8MB buffered Western Digital Caviar WD1200JB is the desktop performance
champion of ATA drives, but IBM 120GXP is the best performance/price choice at
the present time. Indeed, 4x40GB hard drives have too much capacity than we
need (4x10GB is enough) and cost more, but the IBM 120GXP series starts its
capacity from 40GB and it is 20-30% faster than its ancestor, 75GXP series,
which have lower capacity.
Software Choice: Windows NT is the best choice for overlapped I/O programming and that is
why it was the choice of all of the previous PennySort winners. Programming on
Linux is also possible now, since its new kernel 2.4.x begins to support files
larger than 2GB. Considering save the cost of system software, which account
for about 10% of the whole system cost, we chose Redhat
7.2 as our operating system.
At last, we put up a system that
use AMD Athlon XP 1700 CPU, 1GB DDR 266 memory, and 2
RAID 0 disk arrays, running Redhat 7.2 Linux
operating system. Since we bought our components directly from a big
electronics market near our campus and assemble it by ourselves, the cost
composition of our system is very different from previous winners. To stand on
the common ground as other entrants, we try to find lower prices at http://www.pricewatch.com, and manage to
locate all of our parts on only one website, which at last come out to be http://www.ussa.com. After adding $15 assembly
fee and $35 shipping fee that we actually haven’t, the total cost of Table 1 is
almost the same as our real cost.
Table 1: Price Breakdown of 2002
PennySort Machine
|
Quantity |
Description |
Cost |
Total |
|
1 |
Abit KR7A-RAID MOTHERBOARD
RETAIL |
$117.00 |
$117.00 |
|
4 |
IBM 40GB HDD ATA100 IDE 7200RPM OEM |
$71.00 |
$284.00 |
|
2 |
GENERIC PC2100 512MB DDR for Via Chipset MEMORY |
$110.00 |
$220.00 |
|
1 |
AMD ATHLON XP1700 RETAIL BOX CPU |
$135.00 |
$135.00 |
|
1 |
JATON NVIDIA RIVA 128ZX 8MB SGRAM 128BIT AGP 2X |
$10.00 |
$10.00 |
|
1 |
Teac 1.44 MB Floppy Drive |
$8.00 |
$8.00 |
|
1 |
TRENDWARE 10/100 ETHERNET ADAPTER PCI
TE100-PCIWN |
$5.00 |
$5.00 |
|
1 |
ATX Case, 300W, UL, Middle Tower |
$28.00 |
$28.00 |
|
1 |
Assembly |
$15.00 |
$15.00 |
|
1 |
Shipping |
$35 |
$35.00 |
|
Total |
$857.00 |
||
Assuming the computer has a
lifetime of three years, which is 94,608,000 seconds, we divide it by our
system cost (85700 Pennies), and get our time budget for one penny: 1104
seconds.
In addition to our AMD Athlon XP 1700 system, we built an Intel Pentium 1.6GHz
system too. The latter uses Abit TH7II-RAID
motherboard which adopt Intel i850 chipset that support Rambus
Memory. Rambus has a memory bandwidth as broad as
3.2GB/s, while DDR 266 has only 2.1GB/s, and the Front Side Bus (FSB) of
Pentium 4 runs at a 400MHz clock, while AMD Athlon
only at 266MHz. Apart from enormous I/O operations, the main task of our sort
program are memory operations, and this is why we tested the Intel system which
has much broader memory bandwidth. Sure enough, armed with 1GB Rambus memory and 2 RAID 0 disk arrays, the Intel system
turns to be faster than the AMD system, but, only slightly. We think that AMD Athlon CPU make up its disadvantage in memory bandwidth by
providing three Full x86 decoders (while Pentium 4 has only 1) and performing 9
operations per clock cycle (while the Pentium 4 has 4). Consider the much more
expense on the latter system, we still believe the former one is worth to
report here.
Results: We generated various sized files
using the standard SortGen[4] program (Compiled in
Linux, the SortGen generates exactly 100-byte
records.), and sorted them to find the maximum amount that could be sorted
under the time budget. The results are recorded in table 2.
|
Table 2: 2002 PennySort Times |
|||||||
|
Product |
Time Budget |
Best Time |
Sys |
User |
Total CPU Time |
Sorted GB |
Category |
|
THSort |
1104 |
1098 |
150 |
372 |
522 |
9.8 |
Daytona |
Datamation Sort Results
Having a long history that can be
ascended to 1985, Datamation Sort benchmark is the
origin of all sort benchmarks. Its goal is to find out the fastest way that can
sort 1 million 100-byte records. The rapid updating Datamation
world records show the amazing progress of computer technology: 980 seconds in
1987, and only 0.48 seconds in 2001!
The machines used to evaluate the
Datamation benchmark all seem to be giants compared
to PennySort machines. The winner of last year is a cluster with 64 CPUs. Since
our institute has the experience of building the largest cluster in China, we
hope we may have a chance to build a special cluster for Datamation
benchmark. Presently, we’d like to report Datamation
results of our single node cheap computer – it takes THSort
9.47 seconds to sort 1 million records, and this result still came from the two
passes sort. Although we can treat it as a one pass sort by filling all the 1
million records to memory, we didn’t do so since we don’t want to make an
exception on our program.
Minute Sort Results
Minute Sort benchmark is defined
to test how much data can be sorted in one minute, and the past winners were
all supercomputers too. Last year, 21.8 million records were sorted by a 64
nodes cluster, and we also have a wish to join this competition in the future.
At present, THSort is able to sort 6.3 million
records in 60 seconds.
2002 Performance/Price Sort
Mr. Jim Gray points out that
PennySort is a poor benchmark definition[5]. We agree with this, since
it prevents supercomputers to enter the competition, although supercomputers
seem to have little possibility to win the contest. Suppose we have a
supercomputer that cost 2 million dollars, when divide the number of seconds in
three years (94,608,000) by the system cost, it only has 0.473 seconds to run
its sort program. And from the 2001 Datamation
benchmark we can infer that any supercomputer can only sort 1 million records
in 0.48 seconds at most. Then, can we make a conclusion that our cheap sort
machine is 100 times better than the supercomputer in performance/price, since
we have sorted more than 100 million records for a penny? Definitely no. What
hinders the supercomputer to bring into play, is the method that PennySort used
to calculate performance/price ratio. So, it’s reasonable to give different
computers the same time, say, 1 minute, to sort, and then evaluate its
performance/price ratio. And this is what Mr. Jim Gray called Performance/Price
Sort.
On the other hand, the PennySort
winner in the last three years, Mr. Brad Helmkamp and
Mr. Keith McCready, said “We do not favor the method
proposed of doing one minute of sorting then extrapolating the results.
We have found with our own sort that it is not linear in time.” We agree with
this too, since one minute is not enough for most sort programs to run stably.
Then, how about to use 10 minutes as the time budget to test Performance /
Price Sort instead?
Nevertheless, we’d like to report
our Performance / Price Sort results based on one minute time budget here. The
previous method to calculate Performance/Price sort is3:
(1)
Sort the largest file you can in a minute.
(2)
Divide file size in GB by 60 to calculate GB/s.
(3)
Compute the system price in $ per second (3-year depreciation => system
price divided by 9.5e -7 ).
(4)
Compute the GB/$ sorted by dividing item 2 by item 3.
Since calculation based on one
second is unnecessary while one minute is the basic unit, here we suggest
simplify it as the following steps:
(1) Sort the largest file you can in
a minute.
(2) Compute the system price in $ per
minutes (3-year depreciation => system price divided by 1,576,800)
(3) Compute the GB/$ sorted by
dividing item 1 by item 2.
The real meaning of
Performance/Price Sort is to evaluate how much data can a ‘dollar’ sort in
three years, by testing its performance in one minute. Our Performance/Price
Sort results are shown in Table 3:
|
Table 3: Historical Performance/Price
results. |
||||||
|
year |
MB/sec |
GB/$ |
System |
Sys price (M$) |
CPUs |
|
|
1985 |
0.02 |
0.05 |
M6800 Bitton et al |
0.03 |
1 |
Datamation |
|
1986 |
0.03 |
0.01 |
Tandem Tsukerman |
0.3 |
3 |
Datamation |
|
1987 |
3.85 |
0.05 |
Cray YMP, Weinberger |
7.0 |
1 |
Datamation |
|
1991 |
14.29 |
0.54 |
IBM 3090, DFsort/Saber |
2.5 |
1 |
Datamation |
|
1990 |
0.31 |
0.15 |
Kitsuregawa |
0.2 |
1 |
Datamation |
|
1993 |
1.20 |
0.11 |
Sequent, Graefe |
1.0 |
32 |
Datamation |
|
1994 |
1.72 |
0.16 |
IPSC/Wisc DeWitt |
1.0 |
32 |
Datamation |
|
1994 |
11.11 |
5.25 |
Alpha, Nyberg |
0.2 |
1 |
Datamation |
|
1995 |
28.57 |
2.70 |
SGI/Ordinal, Nyberg |
1.0 |
16 |
Minute/Daytona |
|
1995 |
19.61 |
37.10 |
IBM, Agarwal |
0.05 |
1 |
Minute/Indy |
|
1996 |
100.00 |
15.76 |
NOW, Arpaci-Dusseau |
0.6 |
32 |
Minute/Indy |
|
1997 |
140.17 |
8.41 |
Now 95 , Arpaci-Dusseau |
2.0 |
95 |
Minute/Indy |
|
1997 |
86.21 |
|||||