Pex - Automated Whitebox Testing for .NET : Generating Tests

Generating Tests


We assume that you created a Pex project in Visual Studio.

Tutorial: Generating Tests with Pex on the Command-Line


This this tutorial shows you how to test code from the command-line.
We will generate test cases for the following little class:

public static class StringHelper
{
    public static string ReplaceFirst(string s, char c)
    {
        return c + s.Substring(1);
    }
}

Tell Pex what to monitor


// monitor types in MyLibrary.dll
[assembly: PexInstrumentAssembly(typeof(StringHelper))]

Create a new PexClass


  • On the test-project, click on the 'New Item' menu item
newpexitem.png
  • Select 'New Pex Test (xxx)' where xxx is you test framework. We will name our test class StringHelperFixture:
[TestClass, PexClass]
public partial class StringHelperFixture
{
    [PexMethod]
    public void MyPutTest(int i)
    {
        // add parameterized test here
    }
}

  • Refactor MyPutTest to call StringHelper.ReplaceFirst.
[PexMethod]
public void ReplaceFirst(string value, char c)
{
    // invoke method under test with parameter values
    string result = StringHelper.ReplaceFirst(value, c);
 
    // verify assertions
    if (!String.IsNullOrEmpty(value))
    {
        // first character was replaced
        Assert.AreEqual(c, value[0]); 
        // the rest of the string is not modified
        Assert.AreEqual(value.Substring(1), result.Substring(1)); 
    }
}

Explore through the Command Line


  • Open a command window and move to your build folder
  • Execute the following command (the command line supports a rich set of flags, see Command Line for information).
"%programfiles%\microsoft pex\bin\pex.exe" MyLibrary.Tests.dll
Pex: Program Exploration Framework for .Net
Copyright (c) Microsoft Corporation. All rights reserved.
 
instrumenting... launched Pex x86 Edition
(...)
15:08:55.04> MyLibrary.Tests
  15:08:55.05> MyLibrary.Tests.StringHelperFixture
    15:08:55.08> ReplaceFirst(String, Char)
      [test] ReplaceFirstStringChar_61102_150857_0, System.NullReferenceException: Object reference not set to an instance of an object.
      [test] ReplaceFirstStringChar_61102_150858_1, System.ArgumentOutOfRangeException: startIndexcannot be larger than length of string.
Parameter name: startIndex
(...)
15:08:59.21> Finished execution: 1 explorations, 0 failures, 2 generated tests (1 failures, 0 duplicates), 00:00:07.3567
(...)
  EXPLORATION SUCCESS

You can see that Pex generates 2 tests to cover our method, passing a null string and a string of length 0. This is sufficient to cover the 'buggy' implementation of ReplaceFirst. Later in this tutorial, we will fix the implementation, and see how pex generates new tests to cover the changes.

Using the exploration Reports


The pex command line automatically opened an HTML exploration report. This report contains all the information about the test generation, the explorations and the coverage.
reporttestdetails.png

Fixing the implementation and rerunning Pex


We add some parameter validing in ReplaceFirst and re-execute pex:

public static string ReplaceFirst(string s, char c)
{
   if (String.IsNullOrEmpty(s))
   throw new ArgumentNullException("s");
   if (s.Length == 1)
       return c.ToString();
   else
       return c + s.Substring(1);
}

"%programfiles%\microsoft pex\bin\pex.exe" MyLibrary.Tests.dll
Pex: Program Exploration Framework for .Net
(...)
15:12:44.78> MyLibrary.Tests
  15:12:44.80> MyLibrary.Tests.StringHelperFixture
    15:12:44.83> ReplaceFirst(String, Char)
      [test] ReplaceFirstStringChar_61102_151247_0, System.ArgumentNullException: Value cannot be null.
Parameter name: s
      [test] ReplaceFirstStringChar_61102_151248_1
      [test] ReplaceFirstStringChar_61102_151248_2
        [instrumentation] System.Char.ToString not instrumented
        [instrumentation] System.String.Concat not instrumented
(...)
15:12:49.61> Finished execution: 1 explorations, 0 failures, 3 generated tests (0 failures, 0 duplicates), 00:00:08.0440
940
(...)
  EXPLORATION SUCCESS

The generated tests look as follows (for 100% coverage!):

public partial class StringHelperFixture
{        
    [TestMethod]
    [ExpectedException(typeof(System.ArgumentNullException))]
    public void ReplaceFirstStringChar_61102_151247_0()
    {
        this.ReplaceFirst(((string)(null)), '\0');
    }
    [TestMethod]
    public void ReplaceFirstStringChar_61102_151248_1()
    {
        string s0 = "x";
        this.ReplaceFirst(((string)(s0)), 'x');
    }
    [TestMethod]
    public void ReplaceFirstStringChar_61102_151248_2()
    {
        string s0 = "xx";
        this.ReplaceFirst(((string)(s0)), 'x');
    }
}

Looking at code coverage in the reports


In the reports, you can verify that we actually fully cover ReplaceFirst by clicking on the 'coverage' link, and selecting StringHelper.cs:
replacefirstcoverage.png

See Also

PexInstrumentAssemblyAttribute, PexClassAttribute, TestClassAttribute,PexMethodAttribute
(c) Microsoft Corporation. All rights reserved. pex Wiki Documentation 0.93.50813.0