Pex - Automated Exploratory Testing for .NET

• Downloads
• Get started with Pex
• People
• Papers
• Community 
• Check the FAQ

• The result is a small test suite with high code coverage.
• Pex can be used with Test First or Test After development styles.
• Pex emits tests and mocks for MSTest, MbUnit, NUnit, xUnit.Net,...
• Pex integrates with Visual Studio 2008 enabling Exploratory Test Driven Development.
• Pex builds on top of a managed profiling API, Extended Reflection, which allows to easily build .Net monitoring applications.

More links

• Reference Manual (also, this manual as a single page)
• Samples
• Screencasts

Downloads

• Check the System Requirements,
• Pick your download from the Downloads  section,
• Read the Release Notes 
• Get NUnit, MbUnit or xUnit.Net extensions from http://www.codeplex.com/pex  (learn more about it),
• Get started or read to the tutorial

License

Getting Started

• If this is your first time with Pex, try the tutorial to get started.
• You can also get an overview of what Pex is all about.
• Explore the samples.
• If you have particular scenarios in mind, see how Pex will help you.
• Read the reference manual.
• Print the cheat page.

• Join the community .
• Check out the issue tracker.
• Read the FAQ.
• Read the release notes.
• Download additional extensions.
• Go to home.

People

Project Members

• Nikolai Tillmann  (main contact)
• Peli de Halleux 
• Wolfram Schulte 
• Nikolaj Bjørner 

Past Interns

• Dries Vanoverberghe 
• Saswat Anand 
• Christoph Csallner 
• Thorsten Schütt 

Associated Group

• Foundations of Software Engineering , Redmond

Papers

Research Papers

• Pex – White Box Test Generation for .NET link 
• Nikolai Tillmann, Jonathan de Halleux, Proc. of TAP 2008, the 2nd International Conference on Tests and Proofs, LNCS, vol. 4966, pages 134-153, April 2008.
• Parameterized Unit Testing with Pex (Tutorial) link 
• Jonathan de Halleux, Nikolai Tillmann, Proc. of TAP 2008, the 2nd International Conference on Tests and Proofs, LNCS, vol. 4966, pages 171-181, April 2008.
• DySy: Dynamic Symbolic Execution for Invariant Inference PDF 
• Christoph Csallner, Nikolai Tillmann, Yannis Smaragdakis, Microsoft Research Technical Report MSR-TR-2007-151, November 2007.
• Demand-Driven Compositional Symbolic Execution PDF 
• Saswat Anand, Patrice Godefroid, Nikolai Tillmann, Microsoft Research Technical Report MSR-TR-2007-138, October 2007.
• Unit Tests Reloaded: Parameterized Unit Testing with Symbolic Execution
• Nikolai Tillmann, Wolfram Schulte, IEEE Software , vol. 23, no. 4, pp. 38-47, Jul/Aug, 2006.
• Earlier version as Microsoft Research Technical Report MSR-TR-2005-153 PDF , March 2006.
• Discovering Likely Method Specifications PDF 
• Nikolai Tillmann, Feng Chen, Wolfram Schulte, Technical Report MSR-TR-2005-146, March 2006.
• Mock-object generation with behavior link 
• Nikolai Tillmann, Wolfram Schulte, Proc. of ASE 2006 , pp. 365-368, 2006.
• Parameterized Unit Tests PDF 
• Nikolai Tillmann, Wolfram Schulte, Proc. of FSE/ESEC 2005 . Earlier version as Microsoft Technical Report MSR-TR-2005-64, May 2005.

Other Papers

• More papers can be found on the webpage of the MUTT  project; Pex is the successor of MUTT.
• Related papers of other authors:
• The Practice Of Theories 
• Theories in Practice: Easy-to-Write Specifications that Catch Bugs 

Frequently Asked Questions

Could you give me a brief comparison between Pex vs MSTest unit testing?

• a classic unit test

    // no parameters, using 'closed' unit tests
    [TestMethod]
    public void AddMoney() {
         Account account = new Account();
         account.Add(100);
         Assert.IsTrue(account.Amount == 100);
    }

• a parameterized unit test,

    // adding money, we can write a parameterized unit test here
    [PexMethod]
    public void AddMoney(int amount) {
         Account account = new Account();
         account.Add(amount);
         Assert.IsTrue(account.Amount == amount);
    }

Do you support NUnit, MbUnit and xUnit.net?

Can I mix Unit Tests and Parameterized Unit Tests in the same class?

[TestClass]
class Test {
    [PexMethod] void Foo(int i) {... }
    [TestMethod] void Bar() { ... }
}

Is this a random test generation tool?

Is this a test stub generation tool like the MSTest Unit Test Wizard in Visual Studio?

Does Pex work with TypeMock?

Do I need to use the PexClassAttribute?

Unit Testing

Parameterized Unit Testing

[PexMethod]
void ReplaceFirstChar(string target, char c) {
    
    string result = StringHelper.ReplaceFirstChar(target, c);
 
    Assert.AreEqual(result[0], c);
}

class StringHelper {
    static string ReplaceFirstChar(string target, char c) {
        if (target == null) throw new ArgumentNullException();
        if (target.Length == 0) throw new ArgumentOutOfRangeException();
        return c + target.Substring(1);
    }
}

[TestMethod, ExpectedException(typeof(ArgumentNullException))]
void ReplaceFirstChar0() {
    this.ReplaceFirstChar(null, 0);
}
...
[TestMethod]
void ReplaceFirstChar10() {
    this.ReplaceFirstChar("a", 'c');
}

• Writing Your First Parameterized Unit Test
• Stating assumptions and assertions.
• Generic garameterized unit tests
• The philosophy of parameterized unit tests
• Attribute Glossary

See Also

Automated Exploratory Testing

Exploratory Testing

Input Generation By Dynamic Symbolic Execution

1. Pex determines inputs i such that q(i)=false using a constraint solver. By construction, the input i will take an execution path that we have not seen before. (And initially, this means i can by any input since we haven't seen any execution path yet.)
2. Pex executes the test with the chosen input i, and monitors the execution of the test and the program-under-test takes.
3. During the execution, the program takes a particular path which is determined by all the conditional branches of the program. The set of all conditions that determine the execution is called the path condition, written as the predicate p: I -> {true, false} over the formal input parameters. Pex computes a representation of this predicate.
4. Set q := (q or p), i.e. we record that we have now seen the path represented by p.
5. Go to 1.

• Integers, Floats
• Objects
• Structs
• Arrays, Strings

Code Coverage Analysis

Supported Test Frameworks

• MSTest, Visual Studio test framework,
• Repro Test Framework, each test as a separate project
• Support for NUnit, MbUnit or xUnit.Net can be downloaded from the Pex Extensions Project

Attribute Usage Table

• For VSTS, the ITestContext instance is always null.

Specifying up the test framework

[assembly: PexAssemblySettings(TestFramework = "Repro")]

Microsoft.ExtendedReflection

• The managed assembly has the following namespaces:
• Metadata -- metadata model that can faithfully represent all entities that exist at runtime
• DataAccess -- fast and type-safe accessors to "live" program data
• Interpretation -- interpreter of all MSIL instructions, and symbolic representation of runtime values and entire states by terms
• Reasoning -- interface for symbolic reasoning over MSIL instructions
• Monitoring -- the managed counterpart of the ClrMonitor that receives callbacks from the instrumented code
• Utilities -- a set of general utility classes (collections, for debugging, logging, etc)
• The unmanaged CLR profiler called "ClrMonitor"; it instruments MSIL instructions, i.e. redirecting calls and inserting callbacks

Reference Manual

• License
• Cheat Page
• Introduction
• How Pex Will Help You
• Limitations of Pex
• Getting Started
• Requirements
• Downloads
• Installation
• Creating a Pex Project in Visual Studio
• Generating Tests
• The "Hello World" of Pex
• Test Generation
• Configuring Code Instrumentation
• Test To Code Binding
• Test Generators
• Parameterized Unit Testing
• Generic Parameterized Unit Testing
• Exploration of Existing Unit Test
• Assumptions and Assertions
• Pruning by Assumptions
• Input Generation
• Integers, Floats
• Objects
• Using Types
• Instantiating Existing Classes
• Creatables
• Explorables
• Mocks
• Mocking Methods, Properties, Events, COM Interfaces
• Structs
• Arrays, Strings
• Obtaining Additional Inputs from the PexOracle
• Test Failures
• Allowing Exceptions
• Where do the Tests go?
• Settings
• Testing Internal Types
• Test Generation, Advanced Topics
• Coverage
• Visibility
• Pex Packages, Exploration Packages, Path Packages
• Freezing Current Behavior for Future Regression Testing
• Adding a Copyright to Generated Tests
• Generated Unit Test Attributes
• Crafting Test Names
• Exception Injection
• Debugging
• Source Control Interaction
• Symbols And Sources
• Unit Test Frameworks Integration
• SetUp And TearDown (Exploration Life Cycle)
• Ignoring Explorations
• Generated Tests
• Test Bookkeeping
• Issue Tracking
• Propagation of Satellite Attributes
• Supported Test Frameworks
• Supported Test Languages
• Analyzing Reports
• Errors and Warnings
• Report Auto-Cleaning
• Analyzing Uninstrumented Calls
• Analyzing Exceptions
• Exception Frame Tree
• Exception Table
• Analyzing Coverage
• Coverage Filtering
• Analyzing Generated Tests
• Analyzing Boundaries
• Tools
• Command Line
• Command Line Filters Syntax
• Visual Studio Integration
• Snippets
• Pex Wizard
• Extensibility
• Wizard Extensions
• Research Problems
• Coding Guidelines For Extensions
• Packaging Extensions
• Packages
• Graph Package

Reference

• Attribute Glossary
• All Settings
• Warnings And Errors
• Exit Codes
• Static Helper Classes
• Detecting Pex Installation With Wix

WikiBook

• http://research.microsoft.com/pex/wiki/book.html 

Samples

Requirements

• Visual Studio 2008 Professional or higher,
• Visual C#,
• you can force the installation of the samples by using the 'Custom' selection of features in the installer.

Contents

• Pex samples
• Samples.Pex - samples for Parameterized Unit Testing and Pex' extensibility
• ExtendedReflection samples
• Samples.Tracing - a rudimentary call tracer.
• Samples.Reflection - how to read metadata with the ExtendedReflection library on top of which Pex is built
• Visual Studio Extensibility samples
• Samples.VisualStudio.Fixes - how to get our own fix suggestions into Visual Studio with Pex

More Samples

• DySy, advanced extensiblity sample

Screencasts

• Chunker Screencast

Requirements

• Windows Vista 32bit
• Other versions of Windows should work but are untested
• x64 support is under development (available only inside of Microsoft)
• .NET Framework 2.0
• also supported .NET Framework 3.0 and 3.5

Samples Requirements

• Visual Studio 2008 Professional
• Requires Visual C# and MSTest
• Check in Visual Studio in the Help - About Microsoft Visual Studio dialog.

Pex Extensions Project

• custom test framework support,
• custom language support,
• custom exception filtering,
• custom symbolic checking,
• or even directly 'listening' to the instrumentation events.

• Home Page: http://www.codeplex.com/pex 

See Also

Tutorial

• Tutorial (about 80 pages) 

License

Introduction

• Pex can generate inputs for hand-written Parameterized Unit Tests.
• Pex can try to find bugs in an existing .Net assemblies by using the Pex Wizard to create simple parameterized unit tests automatically.
• Pex uses a constraint solver to systematically generate relevant inputs for the program-under-test. The result is a minimal test suite with maximal code coverage. Pex emits tests and mocks for MSTest.

How Pex Will Help You

If you want to ...

• ... write parameterized unit tests
• you can use Pex in Visual Studio,
• ... test an existing code base,
• The Pex Wizard gets you started by generating a set of parameterized unit tests templates and mock objects. Parameterized tests are not executable by themselves, but Pex will generate traditional tests from them. You can refine the templates by adding customer scenarios and validation code.
• ... leverage an existing test suite,
• You can let Pex run your unit tests and inject faults at various places and times.

Writing Parameterized Unit Tests

• Algorithms over primitive data, arrays of primitive data
• Write parameterized unit tests. more...
• Algorithms over interfaces
• Use Pex' automatic mock object generation. more...
• Algorithms over complex data, e.g. compiler
• Let Pex generate an abstract representation of the data first, then feed it to the algorithm. more...
• Let Pex build instances via custom object creation and data invariants, then invoke algorithm. more...
• Data containers
• Write parameterized unit tests. more...
• Let Pex build instances via custom object creation and data invariants, then invoke a method of the container, and check invariants afterwards again. more...
• Write parameterized unit tests which call different methods of the implementation depending on the parameter values. more...
• Client / Server Architectures in a public environment, e.g. web service client, web service server
• Generate interfaces for web services. more...
• Use Pex' mock objects to simulate other side. more...

Cheat Page

Getting Started

Custom Attributes

using Microsoft.Pex.Framework;
[..., PexClass(typeof(Foo))]
public partial class FooTest {
    [PexMethod]
    public void Bar([PexAssumeNotNull]Foo target, int i) {
        target.Bar(i);
    

Static Helpers

[PexMethod]
void StaticHelpers(Foo target){
    PexAssume.IsNotNull(target);
    int i = PexOracle.DefaultSession.Choose<int>("i");
    string result = target.Bar(i);
    PexValue.Add("result", result);
    PexAssert.IsNotNull(result);

Instrumentation

[assembly: PexAssemblyUnderTest("MyAssembly")] // also instruments "MyAssembly"
[assembly: PexInstrumentAssembly("Lib")]

Public Issue Tracker

Issues

ID		Title	Update Date
1942		[external] Support for TypeMock	12/9/2007 11:48:29 PM
2032		[external] Wizard extensibility	1/5/2008 2:47:46 AM
2047		[external] Add a Raise method for events	1/8/2008 3:54:14 AM
2239		[external] inserting methods in opened .g. files corrupts them	2/5/2008 11:26:37 AM
2036		[external] generating test stubs does not update project references	1/7/2008 12:20:46 PM
2031		[external] Wizard should generate set/get/assert systematically	1/8/2008 2:40:18 AM
2237		[external] Add Assert.Inconclusive on generated tests	3/20/2008 8:38:55 AM
2238		[external] Method that returns bool should have a branch	2/14/2008 5:25:19 PM

MSTest

• http://msdn.microsoft.com/vstudio/ 

Z3

PexClassAttribute

• publicly exported type,
• default constructor
• not abstract

• additional suite and categories,

[TestClass] // VSTS test fixture attribute
[PexClass(Suite = "checkin")] // Pex fixture attribute
public partial class MyTests { ... }

• specifying the type under test,

[PexClass(typeof(Foo))] // this is a test for Foo
public partial class FooTest { ... }

See Also

Test To Code Binding

• for you,
• it expresses explicitly the intent of the developer/tester,
• it can be used through the command line to filter tests to run,
• for Pex,
• focus it's search on the code under test,
• instrument the code under test,
• tell the wizard where to add generated files,
• provide coverage numbers with respect to the code under test,
• for other tools
• e.g. which tests to rerun after code has changed.

Assembly Under Test

[assembly: PexAssemblyUnderTest("MyAssembly")]

Type Under Test

[TestClass, PexClass(typeof(Foo))]
public partial class FooTestClass
{ ... 

• PexAllowedExceptionFromTypeUnderTestAttribute can be used to automatically allow a particular exception type form the type under test.

[assembly: PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException))]

Coverage

Coverage Domains

• user code under test: this is the code under test for the current exploration. This code is included in the computation of the coverage numbers (block counts, etc...) and the source coverage will be rendered in the reports. Usually, this is the product code.
• user or test code: this is code that is part of your projects or tests but that is not tested by the current exploration. This code is not included in the computation of the coverage numbers but the source coverage is still rendered in the reports (with different colors). Usually, this is the test code since you don't care about your test coverage but yet it is interresting to see the coverage of tests.
• system code: any framework or third code for which you do not want to collect coverage data. This code is not included in the coverage numbers and source coloring.

Coverage Units

• blocks, basic blocks,
• method touched, the number of methods that were touched at all.

Sample

    class SystemCode {
        public void Run() {
            Console.WriteLine("I should show NOT show up!");
        }
    }
    class UserOrTestCode {
        public void Run() {
            bool b = true;
            if (b)
                Console.WriteLine("I should show up blue!");
            else
                Console.WriteLine("I should show up gray!");
        }
    }
    class UserCodeUnderTest {
        public void Run() {
            bool b = true;
            if (b)
                Console.WriteLine("I should show up green!");
            else
                Console.WriteLine("I should show up red!");
        }
    }
    [TestClass]
    [PexClass(
        // type under test is in UserCodeUnderTest domain
        typeof(UserCodeUnderTest), 
        Suite = "checkin")]
    // UserOrTestCode is the default domain
    // [PexCoverageFilterType(PexCoverageDomain.UserOrTestCode, typeof(UserOrTestCode))]
    [PexCoverageFilterType(PexCoverageDomain.SystemCode, typeof(SystemCode))]
    public partial class CoverageFilterTest {
        [PexMethod]
        public void CallExcludeMe()
        {
            new UserCodeUnderTest().Run();
            new UserOrTestCode().Run();
            new SystemCode().Run();
        }
    }

See Also

PexMethodAttribute

• must be an instance method,
• must be visible to the test class into which the generated tests are placed according to the Settings Waterfall,
• may take any number of parameters,
• may be generic.

Example

[PexClass]
public partial class MyTests {
     [PexMethod]
     public void MyTest(int i)
     { ... }
}

See Also

The "Hello World" of Pex

using Microsoft.Pex.Framework; 
using Microsoft.VisualStudio.TestTools.UnitTesting; 
 
[TestClass]
public partial class HelloWorldTest {
    [PexMethod]
    public void HelloWorld([PexAssumeNotNull]string value) {
        if (value.StartsWith("Hello")
            && value.EndsWith("World!")
            && value.Contains(" "))
            throw new Exception("found it!");
    }
}

[TestMethod()]
public void HelloWorldString_61204_222711_0_07()
{
    this.HelloWorld("Hello World!");
}

See Also

Assumptions and Assertions

[PexMethod]
public void Test1(object o) {
    // precondition: o should not be null
    PexAssume.IsNotNull(o);
 
    ...
}

[PexMethod]
public void Test2([PexAssumeNotNull] object o)
// precondition: o should not be null
{
   ...
}

See Also

Generic Parameterized Unit Tests

[PexClass]
public partial class ListTest {
    [PexMethod]
    [PexUseGenericArguments(typeof(int))]
    [PexUseGenericArguments(typeof(object))]
    public void AddItem<T>(List<T> list, T value)
    { ... }
}

See Also

Attribute Glossary

By Namespaces

• Microsoft.Pex.Framework
• PexClassAttribute
• PexAssumeNotNullAttribute
• PexAssumeUnderTestAttribute
• PexMethodAttribute
• Microsoft.Pex.Framework.Packages
• PexPackageAssemblyAttribute
• PexPackageTypeAttribute
• Microsoft.Pex.Framework.Using
• PexUseTypeAttribute
• PexUseGenericArgumentsAttribute
• Microsoft.Pex.Framework.Suppression
• PexSuppressUninstrumentedMethodFromAssemblyAttribute
• PexIgnoreAttribute
• PexSuppressUninstrumentedMethodFromNamespaceAttribute
• PexSuppressStaticFieldStoreAttribute
• PexSuppressStackFrameFromNamespaceAttribute
• PexSuppressUninstrumentedMethodFromTypeAttribute
• PexSuppressStackFrameFromAssemblyAttribute
• Microsoft.Pex.Framework.Explorable
• PexExplorableFromMethodAttribute
• PexCrossProductExplorableStrategyAttribute
• PexExplorableFromFactoriesAttribute
• PexExplorableMutatorAttribute
• PexRandomExplorableStrategyAttribute
• PexEffectiveExplorableMutatorsAttribute
• PexExplorableFromConstructorAttribute
• PexExplorableFromConstructorAndSettersAttribute
• Microsoft.Pex.Framework.Settings
• PexSettingsMixAttribute
• PexAssemblySettingsAttribute
• Microsoft.Pex.Framework.Validation
• PexAllowedExceptionFromTypeAttribute
• PexAllowedExceptionAttribute
• PexExpectedExceptionAttribute
• PexExpectedGeneratedValueAttribute
• PexAllowedExceptionFromAssemblyAttribute
• PexAllowedExceptionFromTypeUnderTestAttribute
• PexExpectedImprecisionsAttribute
• PexExpectedTestsAttribute
• PexExpectedCoverageAttribute
• Microsoft.Pex.Framework.Strategies
• PexSearchFrontierDepthFirstAttribute
• PexSearchFrontierBreadthFirstAttribute
• PexSearchFrontierRandomAttribute
• PexSearchFrontierIterativeDeepeningAttribute
• Microsoft.Pex.Framework.Instrumentation
• PexInstrumentTypeAttribute
• PexInstrumentMarkedByAttribute
• PexInstrumentTypesByNameAttribute
• PexAssemblyUnderTestAttribute
• PexInstrumentNamespaceAttribute
• PexSubstitutionAssemblyAttribute
• PexInstrumentAssemblyAttribute
• Microsoft.Pex.Framework.Focus
• PexFocusOnAssemblyAttribute
• PexFocusOnNamespaceAttribute
• PexFocusOnTypeAttribute
• Microsoft.Pex.Framework.Exceptions
• PexAssumptionViolationExceptionAttribute
• Microsoft.Pex.Framework.Creatable
• PexCreatablesAsXmlFromAssemblyAttribute
• PexCreatableAsSingletonAttribute
• PexCreatableByMethodAndSettersAttribute
• PexCreatableByMethodAttribute
• PexCreatableByConstructorAttribute
• PexCreatableByConstructorAndSettersAttribute
• PexCreatableAsXmlAttribute
• PexFieldBindingAttribute
• PexCreatableByClassFactoryAttribute
• Microsoft.Pex.Framework.Sequences
• PexRandomTargetStrategyAttribute
• Microsoft.Pex.Framework.Coverage
• PexCoverageFilterNamespaceAttribute
• PexCoverageFilterTypeAttribute
• PexCoverageFilterMethodAttribute
• PexCoverageFilterAssemblyAttribute
• PexCoverageFilterMarkedByAttribute
• Microsoft.Pex.Engine.TestFrameworks
• ReproTestFrameworkAttribute
• MSTestTestFrameworkAttribute
• DirectTestFrameworkAttribute
• Microsoft.Pex.Framework.Wizard
• PexCodePatternAttribute
• Microsoft.Pex.Framework.Generated
• PexRaisedExceptionAttribute
• PexNotCompilableAttribute
• PexGeneratedByAttribute
• PexInjectedExceptionAttribute
• PexNotReproducibleAttribute
• Microsoft.Pex.Framework.MonitoringAndInjection
• PexInjectExceptionsOnUnverifiableUnsafeMemoryAccessAttribute
• Microsoft.Pex.Framework.Factories
• PexFactoryClassAttribute
• PexFactoryMethodAttribute
• Microsoft.Pex.Framework.Mocks
• PexMockAttribute
• PexServiceIdentifierAttribute

Inheritance Tree

• PexAttributeBase (abstract)
• PexComponentElementDecoratorAttributeBase (abstract)
• PexExplorationPackageAttributeBase (abstract)
• PexDomainAttributeBase (abstract)
• PexUseAttributeBase (abstract)
• PexUseTypesAttributeBase (abstract)
• PexUseTypeAttribute
• PexAssumeAttributeBase (abstract)
• PexAssumeNotNullAttribute
• PexAssumeUnderTestAttribute
• PexSuppressUninstrumentedMethodAttributeBase (abstract)
• PexSuppressUninstrumentedMethodFromAssemblyAttribute
• PexSuppressUninstrumentedMethodFromNamespaceAttribute
• PexSuppressUninstrumentedMethodFromTypeAttribute
• PexExplorableAttributeBase (abstract)
• PexExplorableFromMethodAttribute
• PexExplorableFromConstructorAttribute
• PexExplorableFromConstructorAndSettersAttribute
• PexExceptionValidatorAttributeBase (abstract)
• PexAllowedExceptionAttributeBase (abstract)
• PexAllowedExceptionFromTypeAttribute
• PexAllowedExceptionAttribute
• PexAllowedExceptionFromAssemblyAttribute
• PexAllowedExceptionFromTypeUnderTestAttribute
• PexExpectedExceptionAttribute
• PexSearchFrontierAttributeBase (abstract)
• PexSearchFrontierDepthFirstAttribute
• PexSearchFrontierBreadthFirstAttribute
• PexSearchFrontierRandomAttribute
• PexSearchFrontierIterativeDeepeningAttribute
• PexAssumptionViolationExceptionAttribute
• PexCreatableFactoryAttributeBase (abstract)
• PexCreatablesAsXmlFromAssemblyAttribute
• PexTargetStrategyFactoryAttributeBase (abstract)
• PexRandomTargetStrategyAttribute
• PexExplorableStrategyAttributeBase (abstract)
• PexCrossProductExplorableStrategyAttribute
• PexRandomExplorableStrategyAttribute
• PexCreatableAttributeBase (abstract)
• PexCreatableAsSingletonAttribute
• PexCreatableByMethodAndSettersAttributeBase (abstract)
• PexCreatableByMethodAndSettersAttribute
• PexCreatableByMethodAttribute
• PexCreatableByConstructorAttribute
• PexCreatableByConstructorAndSettersAttribute
• PexCreatableAsXmlAttribute
• PexCreatableByClassFactoryAttribute
• PexExplorableFromFactoriesAttribute
• PexExpectedGeneratedValueAttribute
• PexTestNamerAttributeBase (abstract)
• PexExplorableMutatorAttribute
• PexMonitorFactoryAttributeBase (abstract)
• PexInjectExceptionsOnUnverifiableUnsafeMemoryAccessAttribute
• PexSuppressStaticFieldStoreAttribute
• PexEffectiveExplorableMutatorsAttribute
• PexExpectedImprecisionsAttribute
• PexExpectedTestsAttribute
• PexExpectedCoverageAttribute
• PexFocusOnAttributeBase (abstract)
• PexFocusOnAssemblyAttribute
• PexFocusOnNamespaceAttribute
• PexFocusOnTypeAttribute
• PexCoverageFilterAttributeBase (abstract)
• PexCoverageFilterNamespaceAttribute
• PexCoverageFilterTypeAttribute
• PexCoverageFilterMethodAttribute
• PexCoverageFilterAssemblyAttribute
• PexCoverageFilterMarkedByAttribute
• PexPathPackageAttributeBase (abstract)
• PexLogEventFilterAttributeBase (abstract)
• PexSettingsStoreAttributeBase (abstract)
• PexSettingsAttributeBase (abstract)
• PexClassAttribute
• PexExplorationAttributeBase (abstract)
• PexMethodAttribute
• PexSettingsMixAttribute
• PexAssemblySettingsAttribute
• PexMonitorSettingsAttributeBase (abstract)
• PexInstrumentTypeAttributeBase (abstract)
• PexInstrumentTypeAttribute
• PexInstrumentMarkedByAttribute
• PexInstrumentTypesByNameAttribute
• PexPackageAssemblyAttribute
• PexInstrumentAssemblyAttributeBase (abstract)
• PexAssemblyUnderTestAttribute
• PexInstrumentAssemblyAttribute
• PexInstrumentNamespaceAttributeBase (abstract)
• PexInstrumentNamespaceAttribute
• PexSubstitutionAssemblyAttribute
• PexPackageAttributeBase (abstract)
• PexSuppressStackFrameAttributeBase (abstract)
• PexSuppressStackFrameFromNamespaceAttribute
• PexSuppressStackFrameFromAssemblyAttribute
• PexTestFrameworkAttributeBase (abstract)
• ReproTestFrameworkAttribute
• MSTestTestFrameworkAttribute
• DirectTestFrameworkAttribute
• PexFixItEmitterFactoryAttributeBase (abstract)
• PexLanguageAttributeBase (abstract)
• PexExplorableGuesserAttributeBase (abstract)
• PexSourceControlAttributeBase (abstract)
• PexProcessSourceControlAttributeBase (abstract)
• PexGenericArgumentsProviderAttributeBase (abstract)
• PexUseGenericArgumentsAttribute
• PexCodePatternAttribute
• PexRaisedExceptionAttribute
• PexIgnoreAttribute
• PexFactoryClassAttribute
• PexNotCompilableAttribute
• PexMockAttribute
• PexGeneratedByAttribute
• PexFactoryMethodAttribute
• PexInjectedExceptionAttribute
• PexPackageTypeAttribute
• PexNotReproducibleAttribute
• PexFieldBindingAttribute
• PexServiceIdentifierAttribute

PexAssume

Example

public void TestSomething(int i, int j) {
     PexAssume.AreNotEqual(j, 0);
     int k = i/j;
     ...
}

Remarks

     if (j==0)
          return;

See Also

MbUnit

Test Inputs

• The parameters of a parameterized unit test.
• Values obtained from the PexOracle.
• Whether or not throw exceptions when exception injection is active.

Constraint Solver

Path Condition

  [PexMethod]
  public void Test(int i) 
  {
      if (i>0) 
          Console.WriteLine("positive");
  }

• Test(-1), which can be characterized by the condition i<=0, and
• Test(0), which can be characterized by the condition i>0.

Generating Integers and Floats

Generating Objects

• Instantiating Existing Classes
• Mocks

Generating Structs

Arrays, Strings

PexOracle

Modes of operation

• While Pex is performing a symbolic analysis of the test and the tested code during Input Generation, the oracle returns arbitrary values, and Pex tracks how the value is used in the test and the tested code. Pex will generate relevant values to trigger different execution paths in the test and the tested code.
• The generated code for particular test cases sets up the oracle in a specific way, so that the reexecution of such a test case will make particular choices to trigger a particular execution path.

Usage

• If you want to use the oracle to determine the behavior of a method (e.g. of a mock object), you can call PexOracle.MethodCall, which returns a IPexMethodCallOracle.

public int Foo() {
    IPexMethodCallOracle call = PexOracle.MethodCall(this);
    return call.ChooseResult<int>();
}

• If you want to draw an auxiliary input value (or a sequence of auxiliar input values), you can call PexOracle.CreateSession, which returns a IPexOracleSession.
• You can use the static PexOracle.DefaultSession if a single static session works for you.

public void Test() {
    ArrayList list = new ArrayList();
    while(PexOracle.DefaultSession.Choose<bool>("next"))
        list.Add(null);
    ...
}

See Also

Mocks

The conventional answer: mock objects with explicit behavior

Generated mock objects with generated behavior

• Given an interface or a non-sealed class, Pex can generate source code for a mock class in Visual Studio.
• Given such a stub, Pex can generate the behavior of a mock object just like it determines relevant input values for test parameters of parameterized unit tests.

Choose and Replay

• choosing, when exploring code, Pex mocks are a source of additional test inputs, and Pex will try its best to choose interesting values,
• replay, when executing a previously generated test, Pex mocks behave like stubs with behavior (i.e. predefined behavior).

Example

interface INamed {
    string GetName();
}

Writing the mock

[PexMock] // specifies to Pex that this type is a mock
public class MNamed : INamed {
    public string GetName() {
        IPexMethodCallOracle call = PexOracle.MethodCall(this);
        return call.ChooseResult<string>();
    }
}

    IPexMethodCallOracle call = PexOracle.MethodCall(this);

    return call.ChooseResult<string>();

Let's try it!

[PexMethod, PexUseType(typeof(MNamed))]
public void GetName([PexAssumeNotNull]INamed name) {
    if (name.GetName() == "Marc")
        Console.WriteLine("hello marc");
}

Generated tests

// test 2
MockTest.MNamed mn0 = new MockTest.MNamed();
IPexOracleRecorder oracle = PexOracle.NewTest();
oracle.OnCall(0, "MNamed.GetName()")
      .Returns("Marc");
this.GetName(mn0);

// on the call '0' to the method GetName of MockType.MNamed,
oracle.OnCall(0, "MNamed.GetName()")
// return "Marc"
      .Returns("Marc");

Adding behavior

[PexMock] // specifies to pex that this type is a mock
public class MNamed : INamed {
    private string name;
    public string GetName() {
        if(this.name == null)
            this.name = PexOracle.MethodCall(this).ChooseNotNull<string>("name");
        return this.name;
    }
}

More on mocks

• Methods,
• Properties,
• Events
• COM interface,
• Fault Injection With Mocks

See Also

NUnit

xUnit.Net

• http://www.codeplex.com/xunit 

TestFramework

See Also

PexAssemblySettingsAttribute

using Microsoft.Pex.Framework;
// overriding the test framework selection
[assembly: PexAssemblySettings(TestFramework = "Naked")]

• CoveragePublisher
• TestFramework
• TestLanguage
• TestNoPartialClasses
• TestNoClassAttribute
• TestOverrideReadonly
• TestGenerateDuplicates
• TestRootNamespace
• TestForceFixtureSetupTeardown
• RequiredCoveragePercentile
• NoSeparateAppDomain
• TestCopyright

See Also

• It describes the type system from the point of view of the execution engine, which includes explicit box types and information about memory layouts
• It decodes the instruction stream, and provides a sequence of instructions as well as a basic-block representation
• It decodes signature blobs
• It can perform operations which are normally hidden by the execution engine, e.g. vtable-lookups
• It includes facilities to create types which can be used to instantiate generic types whose type parameters have constraints
• It aggressively caches data, and works around some quirks in the standard reflection implementation
• In the following, the core concepts of the library are presented in more detail.
• Layout
• Metadata entities
• Metadata retrieval
• Metadata factory
• Memory regions

• Execution monitor
• Method body executor

Concepts:

• TermManager
• State
• InstructionInterpreter

• Static callbacks which delegate to a user-provided Execution Monitor.
• Substitutions which emulate the behavior of method calls but drive a user-provided Execution Monitor in addition.

• Utilities.Safe.Collections contains a set of collection types which are safe from instrumentation (and therefore faster). This set of types include many commonly desired collection types which cannot be found in the standard System.Collections.Generic namespace.
• Utilities.Safe.Diagnostics provide alternatives to the Debug and Debugger class of the standard System.Diagnostics namespace.

Overview

• (Static) calls can be redirected. The redirection target may call the original call target.
• Similar to the standard CLR profiling API, the ClrMonitor monitors when a method is entered or exited.
• It also monitors the concrete parameter values that are passed to a monitored method.
• The execution of every instruction can be monitored.
• The control flow is monitored, i.e. the evaluation of all explicit conditional branches, and of all implicit checks that can cause exceptions.
• Every monitored event causes a callback into the core Microsoft.ExtendedReflection assembly, which can dispatch the callbacks to a user-provided Execution Monitor.

Code instrumentation

Substitutions

Setup

COR_ENABLE_PROFILING=1
COR_PROFILER={C1DAAFC4-10C6-4838-9F89-435E2120F5F1}

• which callbacks will be injected?

CLRMONITOR_FLAGS={flags that decide the extend of the instrumentation}

• which type and assemblies should be instrumented?

CLRMONITOR_INSTRUMENT_TYPES={comma-separated list of type names; case matters; generic types must end in '`n' annotations; nested types are separated by '+'; <Module> stands for all module-level members}
 
CLRMONITOR_INSTRUMENT_ASSEMBLIES={comma-separated list of display names of assemblies; case does not matter; use '*' for all assemblies}
 
CLRMONITOR_INSTRUMENT_TYPES_EXCLUSIONS={list of types which should not be instrumented, despite the above}
 
CLRMONITOR_INSTRUMENT_ASSEMBLIES_EXCLUSIONS={list of assemblies which should not be instrumented, despite the above}
 
CLRMONITOR_DISABLE_MSCORLIB_SUPPRESSIONS=1, disable mscorlib exclusions. Use with *Great* care.

• which types and assemblies should be instrumented so that values can be injected at any place in the execution

CLRMONITOR_INJECT_TYPES={comma-separated list of type names; case matters; generic types must end in '`n' annotations; nested types are separated by '+'; <Module> stands for all module-level members}
 
CLRMONITOR_INJECT_ASSEMBLIES={comma-separated list of display names of assemblies; case does not matter; use '*' for all assemblies}
 
CLRMONITOR_INJECT_TYPES_EXCLUSIONS={list of types which should not be instrumented, despite the above}
 
CLRMONITOR_INJECT_ASSEMBLIES_EXCLUSIONS={list of assemblies which should not be instrumented, despite the above}

• which method calls should be substituted (redirected)?

CLRMONITOR_SUBSTITUTIONS={comma-separated list of file names, pointing to assemblies containing substitutions}

• which instructions should be monitored by inserted callbacks?

Activation

• In a running process, the Monitoring.Controller class provides Start and Stop methods to direct the callbacks to a particular Execution Monitor and to stop the redirection.
• When the following environment variables are set, a user-provided Execution Monitor is loaded automatically into any monitored process. Note that in addition to setting up the environment variables, the Microsoft.ExtendedReflection assembly must either be in the monitored process' application directory, or in the GAC.

CLRMONITOR_USER_ASSEMBLY={file name of assembly containing [Execution Monitor]}
 
CLRMONITOR_USER_TYPE={type in assembly implementing [Execution Monitor]}

Logging (Only available in Debug-builds)

CLRMONITOR_LOGFILE={filename}
CLRMONITOR_VERBOSE=1

Debugging (Only available in Debug-builds)

CLRMONITOR_BREAK=1

Code Substitutions

• methods implemented by native code (x86, X64)
• method calls going through remoting
• managed code which is not instrumented (e.g. some parts of mscorlib cannot be instrumented)