File and Network I/O using Win32 and .NET API’s on Windows XP™

April 2005

Jim Gray, Peter Kukol

In 2005 we updated the Windows I/O studies in light of changes over the last 5 years. The modern studies use SATA disks, and use both native and unmanaged C++, and managed C# code (running on top of the .NET Runtime).

The basic conclusions of this 2005 study are as follows:

Windows XP using NTFS and software RAID come close to delivering the ‘raw’ hardware performance.
A workstation class machine with dual Opteron CPU’s and one AMD 8131 I/O chip with PCI-X and SATA controllers can sustain a little over 1 GB/sec disk bandwidth (reads and writes).
With a server-class four-way Opteron box and a large SATA disk array it’s possible to achieve over 2.5 GB/sec in disk I/O.
Using 10 Gb Ethernet adapters / media we were able to send around 900 MB/sec 10,000 km (from Pasadena to Geneva) via a single stream using Windows sockets – but when doing disk-to-disk transfers the throughput drops to 550 MB/s (local memory-to-memory transfers have been clocked at over 1 GB/s and local disk-to-disk run at up to 750 MB/s). These numbers all use “ordinary” TCP/IP but reflect “tuning” by Gurus who have selected the right socket options (documented in the research reports).
The faster and the longer the network, the more tuning of the networking parameters is required to maximize performance.
Commodity SATA drives deliver up to 60 MB/s read or write bandwidth and with a PCI-X bus and a SATA controller adding more disks in a ‘striped’ RAID configuration scales almost linearly.

The updated diagram at right summarizes some of these speeds. It is based on the original Riedel 1997 study (below).

Here are the supporting documents:

· A report describing the SATA disk I/O measurements: SataDiskIO.doc (875 KB MS Word file).

· A document explaining the best approaches to doing file I/O using the .NET framework: Sequential File Programming Patterns and Performance with .NET (MS Research Technical Report MSR-TR-2004-136).

· A report discussing some of the findings of our trans-continental GB/sec disk transfer effort: Sequential Disk IO Tests for GBps Land Speed Record (MS Research Technical Report MSR-TR-2004-62).

· A collection of programs (each includes a Visual Studio 2003 project with source code and an x86 executable) to measure disk, network and memory speed (plus a few helper utilities):

1. The DiskSpd program to measure disk bandwidth (DiskSpd.zip).

2. The NetSpd program to measure network bandwidth (NetSpd.zip).

3. The MemSpd program to measure memory bandwidth (MemSpd.zip).

4. The GenFile program to quickly create large disk files with randomized content (GenFile.zip).

5. The DumpFile program displays sections of arbitrarily large files in hex and/or ASCII (DumpFile.zip).

Study of Random and Sequential IO on Windows 2000™

Leonard Chung, Jim Gray, Bruce Worthington, Microsoft,
Robert Horst, 3Ware

In 1997, Riedel, van Ingen, and Gray did a study of sequential IO on Windows NT 4.0 (see 2^nd half of this page). We repeated that study and generalized it to include random IO, IDE as well as SCSI disks, and also RAID and network (CIFS/SMB) IO. The basic conclusions of the study are:

Windows2000 performance is comparable or slightly better than NT 4.0 (some bugs were fixed).
New hardware is 3x faster than the old hardware on sequential.
IDE and SCSI performance are comparable on sequential IO.
It is possible to build a “served” terabyte for $10,000 using IDE drives.
Out of the box, Windows2000 delivers about 40MBps in CIFS/SMB read performance, while write performance is ½ that (one must use gigabit Ethernet to achieve these speeds.)
There are some serious performance problems in the CIFS/SMB software stack.

The RAP/PAP diagram of Riedel et al is revised as show in the diagram at the right:

Here are the supporting documents:

· A report describing our measurements. Win2K_IO_MSTR_2000_55.doc (1.3 MB MSword file) and Win2K_IO_MSTR_2000_55.pdf (400 KB PDF file).

· A presentation of our results Win2K_IO.ppt (605 KB PowerPoint file) and Win2K_IO.pdf (520 KB PDF file).

· The SQLIO2 program as a Visual Studio project and an exe (sqlio2.zip).

· The CashFlush program used in our tests to clear the cache CacheFlush.zip (58 KB).

· The MemSpeed program used in our tests to measure CPU, memory, controller, and disk speed as MemSpeed.zip (43 KB zip file).

· The FileExtendSpeed program used to measure how quickly files can grow FileExtendSpeed.zip (50 KB).

· Spreadsheets containing all our measurements results.zip (5 MB).

1997 Study of Sequential IO on Windows NT 4.0™

Erik Riedel, Catharine van Ingen, Jim Gray, Microsoft Research

High-speed sequential file access is important for bulk data operations typically found in utility, multimedia, data mining, and scientific applications. High-speed sequential IO is also important in the startup of interactive applications. Minimizing IO overhead and maximizing bandwidth frees power to process the data. The goals of this study were twofold:

Understand the basic hardware limits - separating the Peak Advertised Performance (PAP) from the Real Achievable Performance (RAP).
Do better cheaply - to increase sequential IO throughput and decrease processor overhead while making as few application changes as possible.

The report shows that NTFS out-of-the box read and write performance is quite good, but overheads for small requests can be quite high. The best performance is achieved by using large requests, bypassing the file system cache, spreading the data across many disks and controllers, and using deep-asynchronous requests.

PAP (Peak Advertised Performance) vs RAP (Real Achieved Performance)

The paper: (MSWord, 2MB)

The sample programs: (WinZip 19KB)

The program that generated the experimental data: (WinZip 66KB)

The talk Jim gave Dec 1 in Redmond (Gray_ECS_Disk.ppt 1.5MB)

The NetShow of the talk Jim gave in Redmond netshow only available inside of Microsoft.

Note: In December 1999 Tadeusz Wellenger [tadeusz@cs.cornell.edu] reported a bug in the IO completion routine +Event logic. We fixed that and added a IO_CompletionPort example which is the "right" way to do asynchronous IO under NT and Windows 2000.

A recent paper by Bruce Worthington on EIDE performance (IDE Ultra/33 Performance: Intel PIIX4E): (IDE_Performance_Report.doc 280KB) (IDE_Performance_Report.pdf 70KB)

Write us at vanIngen@Microsoft.com, riedel+@cmu.edu