Ben's CLI Benchmarks

The purpose of this page is to document a collection of managed code benchmarks available for performance studies of CLI implementations. 

Please note that I did not write these benchmarks, I just collected them.  If you have other benchmarks that you think should be included in this collection, please let me know (email to zorn at microsoft dot com).  Currently, these benchmarks are available as a binary-only release.  Sources may also be available through other links.

Here are the benchmarks that are currently available:

• lcscbench - compiler benchmark

  • Description: lcscbench is based on the front end of a compiler for C# written in C# and uses a generalized LR (GLR) parsing algorithm.  lcscbench is compute and memory intensive, requiring hundreds of megabytes of heap for the largest input file provided (a C# source file with 125,000 lines of code). The source contains approximately 17,000 lines of code if machine-generated code is not counted.

  • Developers: Dave Hanson and Todd Proebsting (Microsoft Research)

  • Compatibility: CLR 1.0.3705, Rotor 1.0 (others untested)

  • Download: LCSC Managed Code Benchmark or visit http://research.microsoft.com/research/downloads/default.aspx

  • Source availability: currently unavailable
     

• ahcbench - compression benchmark

  • Description: ahcbench is based on compressing and uncompressing an input file using Adaptive Huffman Compression. It is written in C#.  ahcbench is compute-intensive, requiring a relatively small heap. There are 1267 lines of code in the source for the benchmark.

  • Developer: Based on an implementation of Adaptive Huffman Compression by Stephen Toub

  • Compatibility: CLR 1.0.3705, Rotor 1.0 (others untested)

  • Download: AHC Managed Code Benchmark or visit http://research.microsoft.com/research/downloads/default.aspx

  • Source availability: the source code for the implementation can be obtained at http://www.gotdotnet.com
     

• SharpSATbench - satisfiability solver benchmark

  • Description: SharpSATbench is a CLI managed-code benchmark based on a clause-based satisfiability solver where the logic formula is written in Conjunctive Normal Form (CNF). SharpSATbench is compute-intensive, requiring a moderate-sized heap. There are 10,900 lines of code in the C# source of the benchmark.

  • Developer: Lintao Zhang

  • Compatibility: CLR 1.03705, Rotor 1.0 (others untested)

  • Download: SharpSAT Managed Code Benchmark or visit http://research.microsoft.com/research/downloads/default.aspx

  • Source availability: currently unavailable

  • Test input availablilty: many additional .cnf test inputs are available from  http://www.ece.cmu.edu/~mvelev/sat_benchmarks.html

Other C# / CLI Benchmark Resources

• Chandra Krintz's Phoenix-Compatible C# Benchmarks

This page is maintained by Ben Zorn (zorn at microsoft dot com).