Testing a Task

In my previous post I described the creation of a custom MSBuild task to aid in building WiX projects under TFS. However, one thing I have always disliked about most of the code nuggets found on blogs and magazine articles, which I find highly inexcusable in today’s development landscape, is that they rarely include any tests to prove that their code actually works. Therefore, this post is intended to demonstrate that throwing in a few simple unit tests with code samples is easy and goes a long way toward helping readers understand and trust the code being published.

The first thing that needs to be addressed in this case is the fact that the Software Under Test (SUT) is actually an extension for a third-party application – the MSBuild engine. For someone new to unit testing, that may seem like an insurmountable hurdle, after all I certainly don’t want to have to invoke the entire build framework in order to unit test my 188 line custom task. Fortunately some smart people have already shown us that an easy way to deal with those scenarios is to introduce a “test double”, the options and vocabulary of which are well explained by Martin Fowler in this article.

While mock objects enable a deep level of testing and behavior assurance, manually creating them can be tedious depending on the object being mocked and the number of behavioral combinations involved. To help with that, many developers resort to frameworks such as TypeMock, Rhino Mocks, Moq, et al. Of those, TypeMock has the distinction of using aspect-oriented techniques to actually redirect calls from the real code allowing you to test units which are otherwise difficult or impossible to isolate. Fortunately, in this particular case we have some nice interfaces to work against and I’m not especially concerned about guaranteeing internal behavior of my code, so I will make use of a simple stub object and focus on validating known inputs against expected outcomes.

Since the SUT is targeting Team Build I’m going to stay in that arena and use the testing tools built into Visual Studio Team System 2008 Development Edition for this sample. However, the code and concepts could easily be adapted (in many cases with no change) to work in NUnit, MbUnit, or even xUnit.net. All of which can be integrated into Visual Studio either natively or via TestDriven.NET.

For this post I’m only going to include two tests, each of which proves that a valid input produces the expected result. What is missing from a thorough test suite is the slew of exception cases which should also be covered. For instance, what happens when an invalid input is applied, or when the environment doesn’t meet preconditions, etc. However, those tests are actually easier to write and for the sake of brevity I’ll leave them to the reader.

The first thing I did is create a new project in the build task solution called WixVarSubstitutionTests (there are many varied opinions on the pros and cons of separating tests from code, but at Tranxition we’ve decided to adopt that approach so I’m using it here.) Next add a project reference to the WixVarSubstitution project and make sure references are also present for Microsoft.Build.Framework, Microsoft.Build.Utilities.v3.5, and System.Xml.Linq. Finally, create a new C# class called WixVarSubstitutionTaskTest and we’re ready to write some tests.

Now that we’re ready to write some code, we need to build up a bit of infrastructure to enable our test methods. All test “fixtures” need to be adorned with the TestClassAttribute, and we’ll need to initialize a task instance for each test method:

18 [TestClass()]

19 public class WixVarSubstitutionTest

20 {

21   WixVarSubstitution _task;

22

23   [TestInitialize()]

24   public void TestInitialize()

25   {

26     _task = new WixVarSubstitution();

27     _task.BuildEngine = new BuildEngineStub();

28   }

Notice the BuildEngineStub reference? That is a stub object (test double) which we’ll use to isolate our task from the actual MSBuild engine. So the only thing left to do before writing our tests is to create that stub class:

147 private class BuildEngineStub : IBuildEngine

148 {

149   public bool BuildProjectFile(

150     string projectFileName, string[] targetNames,

151     IDictionary globalProperties, IDictionary targetOutputs ) { return true; }

152

153   public bool ContinueOnError { get { return false; } }

154

155   public int ColumnNumberOfTaskNode { get { return 0; } }

156

157   public int LineNumberOfTaskNode { get { return 0; } }

158

159   public string ProjectFileOfTaskNode { get { return string.Empty; } }

160

161   public void LogCustomEvent( CustomBuildEventArgs e ) { return; }

162

163   public void LogErrorEvent( BuildErrorEventArgs e ) { return; }

164

165   public void LogMessageEvent( BuildMessageEventArgs e ) { return; }

166

167   public void LogWarningEvent( BuildWarningEventArgs e ) { return; }

168 }

Obviously, this stub doesn’t do a whole lot. That’s actually the point of a stub, to do as little as possible while providing properties and methods for the supported interfaces. In this case, the only interface we need to implement is IBuildEngine.

One issue you may run into occasionally with stub objects is that the method or property implementation provided causes behavioral issues somewhere down the line. For instance, originally I had the read-only property ProjectFileOfTaskNode return a null reference, however, during testing I discovered that the base Task class doesn’t support that and will throw an exception under certain scenarios. So I simply changed the return to an empty string and the problem went away. Just be aware that this is one of the known limitations with simple stub objects.

Because both of my test methods are going to need a temp file containing some pre-defined XML, I’ll use a helper method to set that up each time:

126 private static string PrepTestFile()

127 {

128   string tempFilePath = Path.GetTempFileName();

129

130   // Create a sample XML document and save it to the temp file

131   new XDocument(

132     new XDeclaration( @”1.0″, @”windows-1252″, “yes” ),

133       new XElement( “Include”,

134         new XProcessingInstruction( “define”, “Var1 = \”Var1Value\”" ),

135         new XProcessingInstruction( “define”, “Var2 = \”Var2Value\”" ),

136         new XProcessingInstruction( “define”, “Var3=\”Var3Value\”" )

137       )

138   ).Save( tempFilePath, SaveOptions.None );

139

140   return tempFilePath;

141 }

The call to Path.GetTempFileName actually creates an empty file in the current user’s temporary files folder and returns the full path as a string. Then a simple XDocument instance is built up using the convenient construction syntax. And finally the constructed XDocument is saved into the new temp.

At last we’ve reached the actual code for our test methods. The first just validates that when a single VariableDefinition is provided, that variable’s value in the XML file is correctly modified:

59 [TestMethod()]

60 public void WixVarSubstitutionTaskTestExecuteWorksGivenOneVariableDefinition()

61 {

62   // Set the SourceFile property to point to the temp file

63   _task.SourceFile = PrepTestFile();

64

65   // Set the VariableDefinitions to effect one of the three variables

66   _task.VariableDefinitions =

67     new XElement( “Root”,

68       new XElement( “VariableDefinition”,

69         new XAttribute( “Name”, “Var1″ ),

70         new XAttribute( “NewValue”, “Var1NewValue” ) )

71     ).ToString();

72

73   // Execute the _task

74   _task.Execute();

75

76   // Load in and process the temp file

77   var e = from n in XDocument.Load( _task.SourceFile, LoadOptions.PreserveWhitespace ).DescendantNodes()

78     where n.NodeType == XmlNodeType.ProcessingInstruction

79           && ( (XProcessingInstruction) n ).Target == “define”

80     select n as XProcessingInstruction;

81

82   // Verify that Var1 was changed as expected

83   // Using a regex to ignore insignificant whitespace

84   Assert.IsTrue( Regex.IsMatch( e.First( pi => pi.Data.StartsWith( “Var1″ ) ).Data, “Var1\\s*=\\s*\”Var1NewValue\”" ) );

85 }

That test follows the common four-stage testing pattern:

1. Set up prerequisite objects
2. Call the method being tested
   _task.Execute()
3. Evaluate the results
   Load the temp file as an XDocument, retrieve the Processing Instructions, and assert that the value for the “Var1″ variable was changed.
4. Tear down the objects

The implicit step in there is #4, obviously, since we’re just allowing the garbage collector to handle object tear-down. It would probably be preferable to delete the temp file explicitly rather than letting it hang around after the test ends.

Now we have a test for providing a single variable definition, the next test will do the same but for providing multiple definitions:

87 [TestMethod()]

88 public void WixVarSubstitutionTaskTestExecuteWorksGivenMultipleVariableDefinitions()

89 {

90   // Set the SourceFile property to point to the temp file

91   _task.SourceFile = PrepTestFile();

92

93   // Set the VariableDefinitions to effect two of the three variables

94   _task.VariableDefinitions =

95     new XElement( “Root”,

96       new XElement( “VariableDefinition”,

97         new XAttribute( “Name”, “Var1″ ),

98         new XAttribute( “NewValue”, “Var1NewValue” ) ),

99       new XElement( “VariableDefinition”,

100        new XAttribute( “Name”, “Var3″ ),

101        new XAttribute( “NewValue”, “Var3NewValue” ) )

102    ).ToString();

103

104  // Execute the _task

105  _task.Execute();

106

107  // Load in and process the temp file

108  var e = from n in XDocument.Load( _task.SourceFile, LoadOptions.PreserveWhitespace ).DescendantNodes()

109    where n.NodeType == XmlNodeType.ProcessingInstruction

110          && ( (XProcessingInstruction) n ).Target == “define”

111    select n as XProcessingInstruction;

112

113  // Verify that Var1 and Var3 where changed as expected

114  // Using a regex to ignore insignificant whitespace

115  Assert.IsTrue( Regex.IsMatch( e.First( pi => pi.Data.StartsWith( “Var1″ ) ).Data, “Var1\\s*=\\s*\”Var1NewValue\”" ) );

116  Assert.IsTrue( Regex.IsMatch( e.First( pi => pi.Data.StartsWith( “Var3″ ) ).Data, “Var3\\s*=\\s*\”Var3NewValue\”" ) );

117 }

And of course writing tests is not particularly useful if you don’t ever run them:

There are lots of additional tests needed, so I’ll just throw out a few obvious ones:

• File is unaffected when one/multiple VariableDefinitions are supplied but no variable names match
• Expected result when SourceFile is marked readonly
  • This actually turned up a problem for us with the custom task which had to be resolved
• “” when SourceFile doesn’t exist
• “” when SourceFile doesn’t contain a valid XML document

It may seem like it took a while getting there, but most of the infrastructure is actually provided for you by Visual Studio Team System if you right click on a class or method and select “Create Unit Tests…”. With a little practice, going from zero to 70% (or more) code coverage should take less than 15 minutes for most units of code. If it takes longer it may be worth refactoring your code or consider adopting a mocking framework to help take care of some of the manual set up.