Sort Benchmark Home Page


Hosted by Microsoft. Webmaster Jim Gray (gray@microsoft.com)

 

 

Special update for 2007:

 

The entry submission date has been extended until 1 May 2007 due to Jim Gray's on-going absence.

 

A provisional committee of past sort benchmark winners has been formed to maintain this year's contest. Members include:

 

Please send (or resend) your entry to vaningen@microsoft.com.

 

 

Year 2006 Results
commentary by Jim Gray on 2006 winners (MSword)

 

Daytona

Indy

Penny

(new 2006) 344 million records (32 GB)
in 1,679 seconds
Bytes-Split-Index Sort (BSIS)
$760 system
1.8 GHz AMD, 1 GB RAM, 4x80GB SATA disks, WindowsXP
Xing Huang and BinHeng Song
School of Software, Tsinghua U., Beijing, China
Bo Huang
Math&CS, Hunan U. of Technology, Zhuzhou, China

(2006)590 M records ( 55GB)
in 644 seconds 
GpuTeraSort
1,469$ system
3 GHz Pentium IV, 2 GB RAM, 7800GT Nvidia graphics card, 9x80GB SATA disks (4 data and 5 “runs”) WindowsXP
Naga Govindaraju, Ritesh Kumar ,
Dinesh Manocha, Jim Gray
U. North Carolina at Chapel Hill, USA 

Minute

(new 2006)  40 GB (400 million records)
NeoSort pdf MSword
Windows, Fujitsu 32 Itanium2, 128 SAN disks
Chris Nyberg, Charles Koester Ordinal Technology

( 2005) 116GB (125 M records)

SCS pdf 58.7 seconds

Linux, 80 Itanium2, 2,520 SAN disks

Jim Wyllie, IBM Almaden Research

TeraByte

(2004) 33 minutes
Nsort pdf, word, htm
Windows, 32 Itanium2, 2,350 SAN disks
Chris Nyberg, Charles Koester Ordinal Technology

(2005) 435 seconds (7.25 minutes)
SCS pdf 

Linux, 80 Itanium2, 2,520 SAN disks
Jim Wyllie, IBM Almaden Research


























 

 

 

Data generator: SortGen.CPP is the source; SortGen.exe is the Windows/Intel binary (uses stdio.h in C): SortGen.zip
is a Visual Studio project. Jim finally fixed it to handle more than 2 billion records. This was not an issue till 2004.

Manual: SortGen #Records FileName

Validator: chkSort.cpp is the source, chkSort.exe is the Windows/Intel binary: chkSort.zip

Manual: chkSort InputFile SortedFile  

checks that files have same checksum, same number of records, and that SortedFile is sorted in strnicmp() order.

Process:

·         Trophies are awarded each year at ACM SIGMOD. 

·         Entrants can contact any previous winner and get their result "certified" by April 1.

·         Entries must include a document describing the algorithm and the hardware in enough detail so that others could reproduce the result. 

                   

Frequently asked questions: FAQ

 

Ground rules: 

·         Must sort to and from operating system files on secondary storage. 

·         No raw disk benchmarks allowed since we are trying to test the IO subsystem. 

·         File or device striping (RAID 0) are allowed (encouraged) to get bandwidth. If file striping is used then the concatenated files must form a sorted file.

·         The output file must be created as part of the sort.

·         Time includes the launching of the sort program. 

 

Original Sort Benchmark “A Measure of Transaction Processing Power ” With 25 others Datamation, V 31.7, April 1985, pp 112-118.  

·         Sort a million hundred-byte records

·         It tests file system, IO system, and utilities. Originally, winners took 1 hour, now 1 second!  So the benchmark is deprecated.

 

Penny and Minute Sort benchmarks RTF format, and HTML

Minute Sort:

·                     Sort as many records as you can in a minute

·                     Report rate and price ($cost/1e6)

Penny Sort:

·                     Sort as much as you can for a penny.

Two categories:

·                     Daytona (stock car): Sort code must be general purpose.

·           Indy (formula 1): need only sort 100-byte records with 10-byte keys.

 

 

Past Winners

commentary by Jim Gray on 2005 winners

 

Daytona

Indy

Penny

Year

Size

System/Group

Year

Size

         System/Group

2005

 

  2004

 

 

 

 

 

2000

  

 

1999

 

1998

15 GB (163 M records)
10 GB

 

 

 

 

 

4.5 GB 

  

 

2.6 GB

 

1.3 GB

PostManSort, doc pdf
 979 sec on a $951 Wintel 2 SATA  
Robert Ramey

THsort pdf, doc
(105 million records)
1098 seconds on a $857 Linux/AMD
Peng Liu , Yao Shi, Li Zhang, Kuo Zhang, Tian Wang, ZunChong Tian, Hao Wang, Xiaoge Wang
Tsinghua University, Beijing, China

HMsort: doc (74KB),pdf (32KB).
Brad Helmkamp, Keith McCready,
Stenograph LLC

HMsort:
  Brad Helmkamp, Keith McCready,
Stenograph LLC

PostmanSort/NT 
Robert Ramey Software
http://www.rrsd.com/

2005

 

 

2002

 

 

 

2000

 

 

1999

 

1998

40GB (433 M records)



12 GB

 

 

 

4.5 GB

 

 

2.6 GB

 

1.5 GB

SheenkSort.pdf
1541 seconds on a 614$ Linux/AMD system
Lei Yang, Hui Huang Zheng Wan, Tao Song
Tsinghua University, Beijing, China

 

125 m records (~12GB)
1380 seconds on a $672 Linux/Intel system
DMsort
Araron Darling, Alex Mohr,
U. Wisconsin, Madison

 

 

 

HMsort:
doc (74KB),pdf (32KB).
Brad Helmkamp, Keith McCready,
Stenograph LLC

886 seconds on a $1010 Win2K/Intel system

 

HMsort:

Brad Helmkamp, Keith McCready,
Stenograph LLC

NTsort /NT
Microsoft
PennySort.doc

Minute

Year

Size

System/Group

Year

Size

System/Group

2004

 

 

 

   2000

 

1998

1997

1995

32 GB

 

 


  12 GB

 

5.8 GB

3.5 GB

1.1 GB

340 million records
Nsort pdf, word, htm
Windows, 32 Itanium2, 2,350 disks
Chris Nyberg, Charles Koester Ordinal

Ordinal Nsort
SGI 32 cpu Origin     IRIX  

 Ordinal Nsort
SGI 32 cpu Origin     IRIX 

Alpha/Nyberg

SGI/Nyberg
Nsort/Irix/Challenge

2004

 

 

2000 

 

1999

 

1998

1997

1995

32GB

 

 

21.8 GB 

 

10.3 GB

 

8.41 GB

3.5 GB

1.08 GB

Nsort pdf, word, htm
32 x Itanium2 WinServer
Chris Nyberg, Charles Koester
Ordinal Technology

NOW+HPVMsort 64 nodes WinNT
 pdf (170KB).
Luis Rivera , Xianan Zhang, Andrew Chien UCSD Y2000..Y2003 winner

NOW+MPI HPVMsort
Luis Rivera UIUC and Andrew Chien UCSD

UC Berkeley NowSort
95 UltraSparc + MyrinetSolaris 

Alpha/Nyberg

SGI/Nyberg
Nsort/Irix/Challenge

Terabyte

year

time

Group

Year

Speed

System/Group

2000 

 

1998

49 min

 

151 min

Tandem FastSort
 Daivd Cossock , Sam Fineberg,
  Pankaj Mehra , John Peck
68x2 Compaq Tandem Sandia

Nsort
Chris Nyberg
SGI   32xOrigin 2000

2000

 

 

1998

18 min

 

 

151 min

SPsort
 1952 SP  cluster 2168 disks
 Jim Wyllie   
1952 SP  cluster 2168 disks  1057 seconds

Nsort
Chris Nyberg
SGI   32xOrigin 2000

 

Datamation

 

 The Datamation benchmark is now deprecated because it is so easy to sort a million records.

Year

Speed

System/Group

1987

1988

1993

1994

1996

1997

1999

  

2000

 

 

 

 

2001

980sec

28 sec

9 sec

7 sec

4.2 sec

2.4 sec

1.18 sec

  

.998 sec

 

  

 

 

.44 sec 

Tandem/Tsukerman

Cray1/Weinberger

Alpha/Nyberg

Alpha/Nyberg

SGI/Nyberg

UltraSPARC/Arpaci-Dusseau

Phillip Buonadonna, Spencer Low, Josh Coates,
UC Berkeley
Millennium Sort 16x2 Dell NT  Myrinet  

Mitsubishi DIAPRISM Hardware Sorter 
HP 4 x 550MHz Xeon PC server + 32 SCSI disks, Windows NT4
Shinsuke Azuma, Takao Sakuma, Tetsuya Takeo, Takaaki Ando, Kenji Shirai
Mitsubishi Electric Corp.
(doc 703KB) or (pdf 50KB)

 Datamation 2001: A Sorting Odyssey,
Florentina Popovici, John Bent, Brian Forney, Andrea Arpaci Dusseau, Remzi Arpaci Dusseau
NOW-sort on 32 Linux PCs
(2xP3(550 MHz), 1 GB, 5x9GB disks). 
 

Warning! Some links in this area let you leave Microsoft's site. The linked sites are not under Microsoft's control and Microsoft is not responsible for the contents of linked sites, or any links contained in a linked sites, or any changes or updates to such sites. Microsoft provides these links only as a convenience. The inclusion of a link does not imply endorsement by Microsoft of the site.