1 comments
.NET 
Last few weeks I received quite a few complaints from people that it was quite impossible to run automated tests on silverlight applications. You can create unit-tests, but running them from a nightly build is impossible, since you can’t run them using a nant script or MsBuild task.
Well, that is about to change. I created a utility that can run your silverlight tests from the console and report the results back in XML to the nightly build. Having said that I have to add a word of warning here. Don’t try this at home!! (at least, building the utility part).
How to use the tool
Before I dive into the tool itself I’d like to explain a little on how you can use the tool in your own projects. You can download the utility at the bottom of this post.
What you need is slunit.exe and a modification to your silverlight unit-test project. Instead of doing a basic setup of the project you will need to modify the App.xaml.cs file to contain the following code:
private void Application_Startup(object sender, StartupEventArgs e)
{
var settings = UnitTestSystem.CreateDefaultSettings();
settings.LogProviders.Add(new HtmlLoggingProvider());
this.RootVisual = UnitTestSystem.CreateTestPage(settings);
}
This will enable the test harness to report the testresults in a special HTML div element at the bottom of the page. This div element will be picked up by the test tool when you run the unit-tests.
That is all that you need to do to enable the use of slunit.exe in combination with your unit-test project. Running slunit.exe is pretty straightforward:
slunit.exe /path:<path to the unit-test host project> /startpage:<name of the testpage> [/timeout:300] [/out:test_results.xml]
The timeout parameter and the out parameter are optional. The above sample shows the defaults for these parameters. You can use this to customize the test process a little according to the needs of your specific situation.
The output of the tool looks similar to the following snippet:
</p> <p><?xml version="1.0" encoding="utf-16"?> <br /><TestResults > <br />&nbsp; <TestResult p2_class="SampleUnitTest" p2_method="FailiingTestCase" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; p2_outcome="Failed" p2_started="2009-05-17T15:57:21.3679241+02:00" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; p2_finished="2009-05-17T15:57:21.3879241+02:00" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /> <br />&nbsp; <TestResult p2_class="SampleUnitTest" p2_method="SampleTestCase" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; p2_outcome="Passed" p2_started="2009-05-17T15:57:21.5879244+02:00" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; p2_finished="2009-05-17T15:57:21.6479244+02:00" <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /> <br /></TestResults></p> <p>
How does it work?
Unit-testing silverlight applications can’t be done using MSTest or NUnit because it isn’t .NET as we know it. The runtime is different in a way that it isn’t standalone, but instead relies on a browser to host it. Because of this limitation you have to run the unit-tests in the browser, since you can’t switch runtimes.
To get around this problem I had to build two core components:
- A client that emulates the browser and is able to report the unit-test results.
- A server that hosts the webapplication containing the unit-tests.
The latter component is not really necessary, because you can also use a basic HTML page to host the unit-test runtime. Except when you need WCF services in your silverlight tests. In that case it is required to have an ASP.NET capable webserver.
The webserver part
The .NET framework comes with the Cassini ASP.NET development server. This webserver is excellent for hosting simple websites and fits the job description for my test host perfectly. The only thing I needed to do is build a basic wrapper around it to fire it up and close it.
The client part
Building the client proved more difficult, because of the fact that I can’t have user interaction with automated processes. I used the standard WebBrowser class from Windows Forms as my client and that proved to be a little problem. To get it fired up I had to hack a little piece of code together that does load the browser, but doesn’t show it to the user.
The following snippet shows how the UnitTestRunner class fires up the browser and navigates to the testpage defined by the user.
WebBrowser browser = new WebBrowser();
_hiddenBrowserForm = new Form() { Width = 20, Height = 20, Visible = false };
_hiddenBrowserForm.Load += (sender, e) =>
{
_testResultsTimer = new System.Threading.Timer(OnTimerCompleted,
null, TimeSpan.FromSeconds(1), TimeSpan.FromMilliseconds(100));
};
_hiddenBrowserForm.Controls.Add(browser);
// Show and hide the browser form.
// This trick is required for the application to finish the test run.
// Otherwise the whole application will freeze on the Application.Run() call.
_hiddenBrowserForm.Show();
_hiddenBrowserForm.Visible = false;
_browser = browser;
browser.Navigate(_targetUri);
Application.Run();
if (TestRunCompleted != null)
{
TestRunCompleted(this, EventArgs.Empty);
}
The code for running the browser has to run in a separate thread. The primary reason for this is that Internet Explorer (Which is what we are essentially running here) requires a STA thread to work. Normal console applications written in .NET don’t work on a STA thread, so we need to simulate that behavior.
This piece of code solves the problem how to get the tests started, but I won’t get any results in the output. It’s impossible to see when the testrun is completed and what the results are. This is caused by the fact that silverlight is sandboxed and so is the Internet Explorer control (in a sense). The only way to get to the test results is by letting the silverlight unit-test harness report its results in a specific div element on the page. This div will then be tracked down by the test runner.
The following piece of code shows how the extraction of the test results works:
_testResultsTimer.Change(Timeout.Infinite, Timeout.Infinite);
// The trick here is to pass control back over to the main “application” thread
// otherwise the webbrowser control is going to freak out on us.
// This call is synchronous, so no need for wait handles etc.
_hiddenBrowserForm.Invoke(new Action(() =>
{
if (_browser.ReadyState == WebBrowserReadyState.Complete)
{
HtmlElement element = _browser.Document.GetElementById(“silverlightTestResults”);
if (element != null)
{
_testResults = element.InnerText;
Application.Exit();
return;
}
}
}));
// Let the test run drop dead when too much time is spend waiting for test results
if ((DateTime.Now – _startTime).TotalSeconds >= _maxDuration)
{
_timeoutExpired = true;
_hiddenBrowserForm.Invoke(new Action(() => Application.Exit()));
return;
}
_testResultsTimer.Change(TimeSpan.FromMilliseconds(100),
TimeSpan.FromMilliseconds(100));
The tracking process works with a timer that checks a predefined div element in the output document of the browser control. Once it is found the windows forms part of the application is killed and the test results are returned to the application. These results are then saved to an XML file.
There are two tricks here to get the application working and make it a little less prone to trashing the buildserver. The first trick here is that while I have a STA thread for the browser, I’m still using a MTA thread for the timer. Because of that I need to call Invoke on the browser to extract the div I need. Otherwise I’ll end up with a very nice application crash.
To prevent the application from trashing the buildserver I added a timeout mechanic that kills the testrunner when it fails to extract the testresults within the specified timeout period.
Reporting the results
Having completed the server and client parts of the solution I needed some way to report the test results in a div element on the html page that is hosting the silverlight unit-test project. Luckely Microsoft thought about it a little and added extension points in the unit-testing framework, where you can plug in your own loggers etc. I used this to writer a logger that gathers the test results and builds an XML structure that is appended to a div element on the page when the test harness completes.
The logger itself listens for two kinds of messages from the test harness. The first is a message indicating the completion of a single testcase. A log message indicating a testcase completion event is recognized by two decorators that have been placed on the message. The following snippet shows how these kind of messages can be recognized:
/// <summary>
/// Checks if the specified message indicates completion of a testcase
/// </summary>
/// <param name=”message”></param>
/// <returns></returns>
private bool IsTestCaseCompletionMessage(LogMessage message)
{
return message.HasDecorators(LogDecorator.TestOutcome,UnitTestLogDecorator.ScenarioResult);
}
Logging the test results is done using the following method:
/// <summary>
/// Reports the test results of a single testcase
/// </summary>
/// <param name=”logMessage”></param>
private void ReportTestCaseResults(LogMessage logMessage)
{
TestOutcome result = (TestOutcome)logMessage[LogDecorator.TestOutcome];
ITestMethod method = (ITestMethod)logMessage[UnitTestLogDecorator.TestMethodMetadata];
ITestClass test = (ITestClass)logMessage[UnitTestLogDecorator.TestClassMetadata];
ScenarioResult sr = (ScenarioResult)logMessage[UnitTestLogDecorator.ScenarioResult];
DateTime startTime = sr.Started;
DateTime endTime = sr.Finished;
XElement testResultElement = new XElement(_namespace + “TestResult”);
XAttribute startedAttribute = new XAttribute(_namespace + “started”, startTime);
XAttribute finishedAttribute = new XAttribute(_namespace + “finished”, endTime);
XAttribute classAttribute = new XAttribute(_namespace + “class”, test.Name);
XAttribute methodAttribute = new XAttribute(_namespace + “method”, method.Name);
XAttribute outcomeAttribute = new XAttribute(_namespace + “outcome”, result.ToString());
testResultElement.Add(classAttribute);
testResultElement.Add(methodAttribute);
testResultElement.Add(outcomeAttribute);
testResultElement.Add(startedAttribute);
testResultElement.Add(finishedAttribute);
_rootElement.Add(testResultElement);
}
The method to report the completed test results is pracically the same. It consists of a method that checks if the incoming message is of the correct type and a method that reports the resutls.
That leaves me with just two lines of code in the constructor of the logger to get the whole thing working:
this.RegisterConditionalHandler(IsTestCompletionMessage, ReportTestFixtureResults);
this.RegisterConditionalHandler(IsTestCaseCompletionMessage, ReportTestCaseResults);
Tips and tricks
I haven’t told the whole story behind the tool. Here’s a list of things that you need to keep in mind when trying to compile the code yourself:
- Don’t ever change the build configuration. The console application needs to be build specifically for x86, otherwise people with x64 machines will have a problem running the tool. Silverlight will not work when the tool is build on x64 or Any CPU. The .NET framework will switch the internet explorer version to x64 when it runs on x64 if you do that. Silverlight wasn´t build for that 😉
- There are no unit-tests for the project, simply because I was unable to come up with a solution for the test-runner to be tested with a silverlight unit-test project attached to it. Instead I preconfigured the debug options of the console app with the right parameters to test it against the provided unit-test sample in silverlight. I know this is manual, but at least it’s something to get stuff going.
Conclusion
This post was probably the longest one up until now and trust me, it toke me quite a bit longer to get my tool working correctly. However I think it’s worth it and I hope people are going to give it a try.
The next step for me is to offer support for testing just XAP packages or even assemblies. This makes it easier to test libraries that don’t need server side components in their unit-tests, but that will have to wait until some other time.
Feel free to e-mail me with questions and/or ideas on how to improve this utility.
One comment
Anyone integrating this into cruisecontrol.net?
Bill