Jonathan Mezach

Throttling workflow services

Jonathan Mezach — Mon, 25 Jan 2010 13:54:01 GMT

For the project I’m currently working on I did some research into the throttling of workflow services using WCF and WF with .NET 3.5. We didn’t have much experience with throttling of services, let alone workflow services so we decided to some simple tests to determine the possibilities and effects of throttling. Before diving into the technical details of how to do throttling I want to look at the scenario we had.

Scenario

In our current implementation we have three different workflows which are all hosted inside the same IIS application. The “Main Workflow” is started through a service operation. It goes on and invokes a couple of different services and at some point it calls a service operation to start “Sub Workflow 1”. This operation is actually called several times and “Sub Workflow 1” performs most of its work asynchronously, so the “Main Workflow” will start up these “Sub Workflows” and then stops.

The implementation of “Sub Workflow 1” eventually makes a call to an external service (thus a different IIS application) and waits for this call to complete. The external service then makes a call back to our workflow service starting yet another workflow (dubbed “Sub Workflow 2”). Schematically this looks like this:

The problem with this approach is that if we were to start up 30 of these sub workflow 1’s we would have all threads blocked waiting for the call to the external service to complete. Because the external service was making a call back to the same application there wouldn’t be enough threads available to process these requests, causing timeouts. Obviously this was not what we wanted.

The first solution we came up with was to split up the workflows into different applications. Logically the workflows should be in one application because they had a strong relation to each other, but we could separate them for technical reasons. So we tested this by creating a new IIS application and rerouting the traffic there. This solved our immediate problem, but we quickly found out that some of the other services couldn’t handle the load of possibly hundreds of workflows making calls to them at the same time. So we decided to look into throttling the workflows so there wouldn’t be more than a fixed number of workflows executing in parallel.

Testing throttling

Out-of-the box WCF already supports the throttling of services. There is a ServiceThrottlingBehavior that you can use to throttle the maximum number of concurrent instances, concurrent calls and concurrent sessions. Unfortunately we couldn’t find any documentation that showed that this would also work for workflow services and the effects that these settings would have on them. So we decide to do a simple proof of concept to see how the throttling settings would affect workflow services.

To test this we made a simple solution which had a Contract project containing the WCF service contracts for both the workflow service and the external service. We also made service agents for both contracts and a UnitTest project where we implemented our scenarios. Finally we made a sequential workflow which looks like this:

We’ve used some custom activities here, so let me explain what this workflow does:

Start a new instance of the workflow by calling “IFirstService.FirstOperation()”.
Synchronously call “ISecondService.SecondOperation()”.
Return the result for “IFirstService.FirstOperation()”.
Asynchronously call “ISecondService.ThirdOperation()” and wait for it to complete.

To make this test as real as possible we implemented a mock for the ISecondService service contract. The mock increases a counter in the SecondOperation call (so we can determine how many workflows were running in parallel). In the ThirdOperation call we used a ManualResetEvent so we could control when they were released. We then determined three scenarios of interest:

What happens if the ServiceThrottlingBehavior is configured to allow a maximum of 3 concurrent instances and we would call the FirstOperation 4 times. We expected that the fourth call would block until we’ve released one of the other three workflows by setting the ManualResetEvent.
What happens in the above scenario if one of the workflows would get suspended? We expected that the fourth call would then be unblocked.
What happens if the suspended workflow from the above scenario was resumed while the other three were still waiting for the result of the ThirdOperation call?

Results

After fiddling around with the tests to create the scenarios described above we found out that the ServiceThrottlingBehavior also works for workflow service, so that was good. In scenario 1 everything worked as we expected. The fourth call was being blocked until we released one of the other three workflows. Once we did that our counter would be increased to 4 so we knew that the fourth workflow actually executed.

Scenario two however didn’t work out-of-the-box. The reason for this was that when a workflow suspends it is not automatically unloaded. This means that the workflow would still be running, thus our fourth workflow didn’t start as there were only three workflows allowed to execute at the same time. But if we unloaded the workflow when it got suspended (meaning it would also get persisted), the fourth workflow would start and everything worked exactly as in scenario one.

The last scenario doesn’t work either. When a request to resume a workflow instance came in while the other three were still executing this request would still be executed, thus four workflows would be running in parallel. This might seem odd, but we reasoned that resuming the workflow instance isn’t done through the WF/WCF integration, but through the underlying WorkflowRuntime directly. We could possibly implement this behavior ourselves, but it might not be important enough to do so.

Conclusion

We concluded that throttling with workflow services basically works how you would expect it to work. I can understand the reason for not immediately unloading a workflow instance when it becomes suspended although it would have been nice if these could be configured somehow. Fortunately we already had a WorkflowRuntimeService in place where we could implement this functionality ourselves.

Creating a workflow image from a workflow service

Jonathan Mezach — Wed, 18 Nov 2009 09:34:21 GMT

Introduction

For the project I’m currently working on we wanted to be able to display an image of a running workflow instance for diagnostic purposes. There are numerous examples of this floating around the internet, but most of them focused on a client side application that would generate the image. This is fine if your client application has access to all the necessary assemblies, but if does not you’re going to run into some problems. In this post I’m going to explain how we solved this problem and some of the pitfalls you might run into.

Scenario

We currently have a fairly large Service Oriented Architecture. We’ve identified a couple of processes that we wanted to support using Windows Workflow Foundation. These processes are implemented as WCF services using Workflow as the actual implementation. There are a couple of these Process Services which each have one or more workflows.

Obviously these Process Services were running in IIS so in order to get an image of a running workflow service we needed some kind of front end application. We already had the infrastructure to resume or terminate suspended workflows from a ASP.Net web application so I decided to extend that with an image of the workflow.

Challenges

I had a look at the Workflow Monitor sample application included in WF and WCF samples for .NET 3.0/3.5. This application basically re-hosts the Workflow designer and pulls data from the SQL tracking store to display which activities have been executed. A nice application, but it didn’t really suit my scenario because that would require my front end web application to have access to all assemblies that my workflow used. This included assemblies containing activities that I used in my workflow as well as the assembly that contained my workflow definition. Not very practical if you have multiple services because I would need to get the assemblies from each of them.

In addition to this, the Workflow Monitor application is a Windows Forms application with interactive controls and such. That didn’t work very well with my ASP.Net web application. And I only needed an image of the workflow anyway and didn’t need any of the interactive stuff. So I had a look around to see if I could find any implementation that would only create the image.

I quickly found this sample. It pretty much did what I needed it to do (and even more using Ajax), but it would still need all of the assemblies of my service in order to work. So I decided to factor out the image creation and put this part on my service and then Stream back the image through WCF to my web application. Sounds simple, right?

Problems

Unfortunately it wasn’t so simple. I quickly ran into an UnauthorizedAccessException. Apparently my service was suddenly trying to access the registry, but it wasn’t allowed to. After some reflecting of het Workflow assemblies I found out that the WorkflowTheme class was the culprit. This class has a static constructor which tries to get the users theme settings from the registry. But my service was running under the Application Pool Identity so it didn’t have a user profile and thus wasn’t allowed to access the registry. Because this is a static constructor it will access the registry as soon as you use that class anywhere in your code, so there wasn’t much I could do about that. But IIS 7.0 (not sure about 6.0) has the option to load a user profile for an application pool and after setting that option, my problem was solved.

Then the next problem came up. In order to provide a visual indication that an activity has executed I had to read from the SQL tracking store. This can be accomplished by using the SqlTrackingQuery class to get a SqlTrackingWorkflowInstance and then reading the ActivityEvents property. This property will execute a stored procedure on the tracking store and then transform each returned row into an ActivityTrackingRecord. This class has a property ActivityType which returns the runtime type of the activity for which that record was created. The tracking store contains both the type name and the assembly name which is transformed to the runtime type while reading the ActivityTrackingRecords. But the default workflow tracking database only has room for a type name of 128 characters. The rest was truncated which resulted in an exception while reading the ActivityEvents property.

I decided to make this column quite a bit bigger because I had a couple of type names that were considerably longer. But then I ran into another problem. To increase performance an index was made on both the type name and the assembly name. Now my index was getting to big which SQL Server wouldn’t accept. In the end I changed the data type of the column from nvarchar to varchar. I don’t think a type name can contain unicode characters anyway and it saves half the space in the database.

Conclusion

It’s nice to be able to re-host the workflow designer and generate images off it. But it doesn’t work very well when the application is not a Windows Forms application. It would have been nice if there was a better way of doing this without depending on the registry or the Windows Forms infrastructure for that matter.

It also seems weird to say that type names would not be longer than 128 characters. With namespaces your type names can get quite long and it even gets worse when using generics.

Even better message based correlation

Jonathan Mezach — Mon, 02 Mar 2009 19:17:31 GMT

As a follow up to my previous post, I've been extending my implementation a bit. In the comments section of my last post you might have read that we were thinking about creating a CorrelationSequenceActivity. I've implemented this activity now and it currently accepts only one child activity which should be a ReceiveActivity. This is validated using an activity validator. It has an attached property called CorrelationKey which will then appear on the ReceiveActivity. You can bind this property to whatever you want in you're workflow. Then when the CorrelationSequenceActivity is executed it registers the correlation key together with the instance ID of the workflow to a runtime service that implements IWorkflowCorrelationRuntimeService. When the CorrelationSequenceActivity is being closed it will automatically unregister the correlation key. We might have to extend this further in the future in order to support a scenario where the workflow is waiting for two events in parallel, each with their own correlation key, but that is not a scenario that we foresee right now.

Building the CorrelationSequenceActivity was quite easy, although I ran into a few problems with the attached property. It isn't as straight forward as using the DependencyProperty.RegisterAttached method. You'll have to implement the IExtenderProvider interface and do all kinds of fancy stuff with designers. Luckily there's already some documentation on how this all works which you can find right here.

So how does this IWorkflowCorrelationRuntimeService interface work? Well, it has three operations, RegisterCorrelationKey, UnregisterCorrelationKey and LookupInstanceIdForCorrelationKey. The first two operations are called by the CorrelationSequenceActivity. The last one is called by the WorkflowCorrelationInstanceContextProvider I talked about in my previous post. So this should take care of correlating the correlation key that was extracted from the incoming message to the right workflow instance. It doesn't solve the extraction of the correlation key from the input message, but it should be theoretically possible to give the CorrelationSequenceActivity another attached property with a designer where you can select a property from the input message and which would then create an XPath query that extracts the key from the input message automagically. As we only have one service operation where we need to do correlation this isn't much of an issue for us right now.

Obviously the IWorkflowCorrelationRuntimeService interface allows for some extensibility. The WorkflowCorrelationEndpointBehavior from my previous post will try to get the configured one from the workflow runtime and if it doesn't find one it will insert a memory based one. This just works with an internal dictionary where the correlation keys and instance ID's are stored. I've also created a SqlWorkflowCorrelationRuntimeService that stores the correlation keys and their associated instance ID's into an extra table in the workflow persistence & tracking store. It implements IPendingWork so it participates in any transaction as needed. This doesn't work correctly when combined with the SharedConnectionWorkflowCommitWorkBatchService, but that's another story.

The only thing I haven't implemented yet is the IEventActivity interface as my colleague Robert suggested. It shouldn't be too hard to implement this though, but we won't need it just yet.

Message based correlation with WF/WCF in .NET 3.5

Jonathan Mezach — Tue, 24 Feb 2009 18:37:36 GMT

When .NET 3.0 was released we got Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF). These were great technologies, but they would have been even greater if they were used together. Unfortunately, Microsoft decided not to include this for .NET 3.0. However, with the release of .NET 3.5 we got the WCF Send and Receive activities that you can use in your workflows in order to implement a WCF service contract by using a workflow. This works great when you want to start a new workflow instance for every call made to your service. But when you want your existing workflow instance to receive the input of another service operation, you'll need to do some correlation.

Microsoft solved this by involving the client. When the first operation that created your workflow instance succeeded, a header or a cookie is attached to the outgoing message containing the workflow instance ID and it is the clients responsibility so store this ID and again attach it to the message when it performs the second operation. This can get tricky if the client is stateless, or when the client that creates the workflow instance is not the same as the client who invokes the second operation. It also requires your client to have knowledge of implementation details of the service. If the client doesn't know that it has to retrieve the instance ID from the reply it gets from the service and then use that same ID again for the second call, it's all going to break apart. This isn't much of a deal if you're developing both the client and the server, but if the client is being developed by a third party it becomes a fair bit more complex.

Ideally you would want the message that is going into the second service operation to have a key value with which you can find the workflow instance back. It would be even better if that value is something that has business value as well, such as an OrderID, or CustomerID. This means you can talk about functional keys outside the service, but you can do the mapping of those functional keys to workflow instances within the implementation of your service.

Unfortunately Microsoft hasn't provided this as an option. It is coming with .NET 4.0, but we're not quite there yet and the programming model for WF 4.0 is going to be quite different when compared to .NET 3.5. We needed a solution for this problem now, rather than having to wait for .NET 4.0 to come out. So I set out to see if we could somehow do message based correlation with the WCF Send and Receive activities in .NET 3.5. I quickly learned that that wasn't an easy task...

The implementation of the WCF Send and Receive activities relies heavily on WCF's extensibility. In fact, Microsoft has made a whole new assembly named System.WorkflowServices which contains a whole bunch of classes that live in the existing System.ServiceModel and System.Workflow namespaces. One of the things they've added is called a context binding. The context binding basically does the whole cookie/header approach to solving the correlation problem. So I thought I'd write a WCF behavior that implements IDispatchMessageInspector that can inspect the messages that are being received at the service, fetch the instance ID from it and then add the required header to the message so that the existing infrastructure from System.WorkflowServices can pick it up and make the call to the right instance. However, it turns out that the message inspectors are almost at the bottom of the foodchain and the whole System.WorkflowServices infrastructure already decided to create a new instance for the second operation before my message inspector was invoked. So that wasn't going to work...

After some more digging around using .NET Reflector I found an interesting class called WorkflowInstanceContextProvider. It was in this class that the message header (or rather a message property that was being fed with information from the header) was being read and the appropriate workflow instance was being attached to the operation context. Unfortunately the class was made internal, so I couldn't inherit from it. It did however implement an interface, which is the standard WCF interface IInstanceContextProvider. So I decided to create my own IInstanceContextProvider implementation that would wrap an existing IInstanceContextProvider implementation, which in this case would be the WorkflowInstanceContextProvider. And to my surprise, this worked perfectly. I made a small test with a workflow that had two Receive activites. When the first one was invoked a CodeActivity made sure that the instance ID of that instance was assigned to a static member in my IInstanceContextProvider implementation. When the second call came in I would inject the value of that static member into the message by using the ContextMessageProperty class. To make my test better I used the input of the first operation as the output of the second operation and that test worked.

All I had to do now was to actually get the value to correlate on from the message, how hard could it be? Well, a bit harder than you would think. The IInstanceContextProvider interface provides to method that both receive the Message instance that is being processed. When you have a Message object you can read it using either the GetBody or the GetReaderAtBodyContents methods. This allows you to read the contents of the message body and do whatever parsing you need on them. The problem is that once you read a message, you can't read the message again. This is done on purpose so you can use streaming messages. If you want to read the message twice you would use the CreateBufferedCopy method and work from there.

This is all good and well, but the interface I was implementing gave me only a reference to the message, but it did not give me a reference to the reference. What this means is that although you can change the message you receive, you can't change the reference to the message so can't create a new message and then assign it to your reference. Well, you can, but whoever called your method still has a reference to the original message. So if I was reading the message in my IInstanceContextProvider implementation it could not be read again further up the WCF stack. Eventually an exception would be raised saying that the message was already read and could not be read again.

So there I was. I had a solution for one problem, but now I had another problem which I couldn't easily fix. That was until my colleague Marcel came up with a simple solution. The Message class I mentioned earlier also has a ToString method. This actually serializes the whole message into XML, regardless of whether it is a streamed or buffered message. When the message is streamed though, you will not get the body of the message, as explained here. When you call ToString you can still read the message later on. So now we have a solution that works, but only if you're using buffered messages. If you use streamed messages things will break. We'll probably implement a check in our implementation that verifies that you are using buffered messages when the behavior is applied, but that's another problem...

Workflows mysteriously being aborted

Jonathan Mezach — Thu, 19 Jun 2008 17:51:46 GMT

Part of the project I'm currently working on involves a workflow system we have build ourselves on top of Windows Workflow Foundation. It consist of a WCF service that contains task, a front end that displays these to users and of course several workflow activities for creating tasks and waiting for them to be completed by the users. Last Friday it was finally time to perform an installation of the system on our acceptance environment so the users could test it. The installation went fine so we were assuming everything was working fine, but yesterday a user reported a problem where she had completed a task, but the next task wasn't appearing in the front end.

Luckily for us we had tracing on on the system, so we investigated the trace log only to find that our workflows were being aborted. We were quite puzzled as to why this was happening, as the system was running fine in our own test environment. So it had to do something with the configuration of the servers. Then I remembered that our colleague Marcel also encountered a problem where workflows were being aborted when he added a SqlWorkflowPersistenceService to the workflow runtime. In his blog he mentioned something about DTC (Distributed Transaction Coordinator) not being set up properly.

As we had problems with MSDTC and Workflow Foundation before we suspected that this was the problem. But we still weren't sure because when it happened before there we're logs all over the place about it. But we couldn't find any in this case. Neither the tracing, the Windows Event Log nor our own log files contained anything even remotely resembling an exception. It was only after I installed WinDbg on the machine in question and attached it to our Windows service that was hosting the WorkflowRuntime that I was able to confirm that it was indeed MSDTC that was causing the workflows from being aborted:

System.Workflow.Runtime Error: 0 : SqlWorkflowPersistenceService(00000000-0000-0000-0000-000000000000): Exception thrown while persisting instance: Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.
System.Workflow.Runtime Error: 0 : stacktrace :    at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
   at System.Transactions.Transaction.Promote()
   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
   at System.Data.SqlClient.SqlInternalConnection.EnlistTransaction(Transaction transaction)
   at System.Data.SqlClient.SqlConnection.EnlistTransaction(Transaction transaction)
   at System.Workflow.Runtime.Hosting.DbResourceAllocator.GetEnlistedConnection(WorkflowCommitWorkBatchService txSvc, Transaction transaction, Boolean& isNewConnection)
   at System.Workflow.Runtime.Hosting.PersistenceDBAccessor..ctor(DbResourceAllocator dbResourceAllocator, Transaction transaction, WorkflowCommitWorkBatchService transactionService)
   at System.Workflow.Runtime.Hosting.SqlWorkflowPersistenceService.System.Workflow.Runtime.IPendingWork.Commit(Transaction transaction, ICollection items)

Normally though, when a WorkflowRuntimeService encounters an exception that it cannot handle, the ServicesExceptionNotHandled event on the WorkflowRuntime is raised. We had an event handler on this event in our implementation which would publish the exception to the log files and also write a trace message containing the exception. We couldn't find any trace of the above mentioned exceptions though, so apparently the SqlWorkflowPersistenceService was handling the exception itself and was silently ignoring it.

So, if you are encountering problems where workflows are being aborted and you can't find out what is wrong, it's possible that MSDTC is to blame. You could then try setting up DTC properly or avoiding DTC altogether by using the SharedConnectionWorkflowCommitWorkBatchService. If this doesn't solve your problems you can try turning up the tracing to be more verbose by following the instructions here.

Service Pack 1 for .NET fixes XML serialization?

Jonathan Mezach — Tue, 10 Jun 2008 20:21:31 GMT

Today we ran into an issue with a Windows Forms application we had written. It was working fine on our development virtual PC's, but for some reason it failed on the actual host machines that we needed it to run on. The application is written purely against the .NET Framework version 2.0. We do communicate with a service that is using WCF, which is part of the .NET Framework version 3.0, but our client side generated proxy was made using wsdl.exe which should produce a client which can run without having WCF. There are various reasons why we can't use .NET Framework version 3.0, but I don't want to go into much detail about that here, except to just say that we can't use .NET Framework version 3.0, and thus cannot use WCF on the client side.

After some debugging we found out that it is actually the server side service that produces an exception. This is because a field that is optional in the Data Contract sometimes becomes required when other fields have certain values. As we couldn't enforce this constraint by making the fields required, we had to check it explicitly on the server side. The weird thing was that the objects that were being sent out by the service were fine. They contained the required property. So it had to be modified on the client side, before being sent back to the server. However, we couldn't find any code that did this in the client. And it was working on our development machines, so why wouldn't it work on the hosts. Weird stuff.

The problem turned out to be a difference between the original .NET 2.0 framework and the .NET 2.0 SP1 version. We had this service pack installed on our images, but not on our host machines. According to KB925272, Microsoft fixed an issue where when the object is being deserialized the fieldSpecified fields weren't set properly. These fields were required because in XML you can either not specify the field at all (so there is no XML element in the serialized XML for the field), or make it xsi:nil="true". However, in our case, once the objects that were being returned by our service got to the client, the fieldSpecified properties were set to false. Then, when the client decides to sent an object back to the server, the value for all properties that had their corresponding fieldSpecified properties set to false wouldn't get to the server. This in turned caused our service to generate an exception, because a field that was required didn't have a value, because it hadn't been sent by the client.

Obviously, this fix is fine by itself. It's just too bad that it breaks existing code. So you'll either have to roll out the Service Pack to all machines, or you'll have to change your code so that this problem cannot present itself (by explicitly setting all the fieldSpecified properties to true, before sending the objects out). We chose the last solution, because if we'd roll out the service pack on all machines we would have to retest all existing applications, because they might be broken after we install it.

Cool tools: NetMassDownloader

Jonathan Mezach — Fri, 23 May 2008 16:43:24 GMT

Today I was working on an issue in our application that was brought to our attention by one of the users. So I started investigating this issue and soon found out that the problem was situated in our ASP.Net front end. While I was debugging and analyzing the call stack I found out that something went wrong between the page that is being posted and the Mediator, which is responsible for actually saving the data to our back end services. We were using a GridView on our page and for some reason the objects that got to the Mediator didn't have its properties set right, so values from one row in the grid were being saved to another row. Unfortunately I couldn't really figure out what went wrong in between.

I then remembered that a while ago Microsoft released symbol files and source code for the .NET Framework. So I figured out that if I could get this, I could step into the code of some of the ASP.NET WebControls and see what was going on there. After a quick Google search I found a patch for Visual Studio that I needed to install and then change some Visual Studio settings and I was good to go. The patch would retrieve the needed source code from Microsoft's server on the fly, as I was stepping through the code. Unfortunately the said patch was only supposed to work for Visual Studio 2008 and we are still using Visual Studio 2005. As upgrading wasn't a real option, I was kinda stuck.

After Googling some more, I found this tool called NetMassDownloader, over at CodePlex. What it basically does is download the necessary .PDB and source code files for a specific assembly, or a directory containing one or more assemblies. The original article I found about this also explained how to set up Visual Studio 2005 to use these. So after downloading the tool, using it to download the PDB and source files for System.Web and setting up Visual Studio 2005 to use them, I was stepping through the GridView code and finally discovered what the problem was. Too bad that the problem was in fact in something I had written, but well, you can't get everything right the first time. I'm sure though that this will come in handy at some point in the future.

Issue fixed

Jonathan Mezach — Wed, 30 Apr 2008 11:28:14 GMT

I thought I'd write a little follow up on my previous post. We found out yesterday that the bug we found in Windows Workflow Foundation which caused us to make a work around, has been fixed in the recently released .NET Framework 2.0 and 3.0 SP1. We haven't been able to find a mention of this bug in the release notes, but we had a small application that showed the bug was there, so we tested it on our development machine which were using the service packs and then on a test server which didn't have the service pack installed and it showed that it was working find on our development machines, but it didn't work on our test servers.

So yesterday we've removed the work around from the custom activity we had and created a patch for the application which can be rolled out to the production environment. We still have to delete all the broken workflows though, because once the currently existing timers go off, the activity will Execute again, and once that happens the users won't be able to use those workflows anymore. But at least the bug is fixed and we don't have to use the workaround anymore.

This doesn't mean however that you can start writing activities that return ActivityExecutionState.Executing from their Execute method when being used in an EventDrivenActivity in a state machine workflow. This will still prevent other EventDrivenActivities in the same state from becoming active when a message is delivered to their queue.

State machine workflows

Jonathan Mezach — Mon, 28 Apr 2008 19:37:11 GMT

At the beginning of last week a report came in from our users which said that they were experiencing timeout issues while working with the application that has been running in production quite stable since the beginning of this year. Because this was a blocking issue we quickly started investigating what went wrong. After looking at the code for about half an hour we decided we needed to put on tracing to find out what really happens. So we did that and we found out that the events we were sending to our workflows weren't delivered. We couldn't quickly find out what the problem was, so we decided to remove the running workflow from the persistence store, which would cause the application to create a new workflow and force it to the correct state. It wasn't the most ideal solution, but at least it meant that the users could continue working with the application. We did however request a backup of the database so we could debug the problem a bit more.

So after I had restored the backup from the production environment on my local development machine and got everything set up I found out that the reason the events weren't delivered to the workflow was because the scheduler was busy executing another activity. It took quite some debugging and reflectoring to find out that when a message is written to the workflow queue, the event doesn't get fired until the SchedulerBusy flag on the StateMachineExecutionState internal class is set to false. But I still couldn't figure out why this specific flag was set to True, instead of False.

A colleague of mine suggested to use the Workflow Monitor, which is one of the applications that comes with the Windows Workflow Foundation samples. It queries on a SQL tracking database and displays the current state of all the running workflows. Quite a nifty tool if you ask me. So I started it up and had a look at the workflows that were working and the workflows that were broken. Although it wasn't obvious immediately, I found out that the workflows that were broken were currently executing a ConfigurableDelayActivity.

This was an activity that was written by my colleague Robert and it was supposed to solve two problems. The first one was that there was a requirement to be able to configure various delays that were used by the workflow. The second problem it tried to solve was a bug that we found in Windows Workflow Foundation. You can find the full details of this bug in the linked post, but in short it meant that if you were using the ManualWorkflowScheduler (which is recommended when hosting in IIS) and you want a timer that expires more than 47 days in the future, you get an exception. So with this activity we thought we had a workaround.

Unfortunately we found out that our work around wasn't working correctly. What was happening is that when the ConfigurableDelayActivity becomes the next activity to execute, it would create a timer queue for itself and set it to some time in the future. When that time in the future was more than the earlier mentioned 47 days away, it would make a timer for 47 days. Then, when the timer went off, the activity started executing, so the Execute method was called. There we determined whether the actual delay has expired. If it hadn't expired yet, we would wait some more by repeating the process. After that, the activity would return ActivityExecutionState.Executing. This is were the problems start.

Because the ConfigurableDelayActivity implements the IEventActivityListener interface, the activity itself isn't really subscribed to the event that gets raised when a message is put in the queue. Rather, the activity hosting the ConfigurableDelayActivity, which would be an EventDrivenActivity gets sent the message, which will in turn call the Execute method on the activity implementing IEventActivityListener. The problem we were having is that once the 47 days timer went off, the activity gets executed and then says it is still executing, which resulted in the scheduler being busy. This blocks any other EventDrivenActivity in the same state from receiving a message on a queue, because the SchedulerBusy flag is set.

Now, this might all be the way it is supposed to work. It's too bad thought that there isn't any documentation on it, or at least we haven't been able to find any. In fact, I had a look at what the out of the box DelayActivity does, and it explicitly checks whether it is running inside an EventDrivenActivity, and if it is, it will always return ActivityExecutionState.Closed. But as this is all internal stuff, how are we supposed to know? So that's why I thought I'd write a blog about it, because I'm sure somebody else will run into these problems.

WF: Required bindable properties

Jonathan Mezach — Wed, 19 Sep 2007 20:08:00 GMT

Today I was working on a bit of workflow we have in our project. We've written a couple of activities to handle some of the processing done in our application (such as sending out emails at a specified point in time). These activities have been working quite well, but they lacked some kind of validation on required properties or how they were used in the workflow. So I was assigned the task to implement some of that stuff.

Windows Workflow Foundation has build in support for doing validations on workflow activities. This is as simple as building a class that extends the ActivityValidator class and then applying an ActivityValidatorAttribute to your custom activity that specifies the class that implements the validation logic. But that is just one way to do it, because there is also something that is called a ValidationOption enumeration, which has the values None, Optional and Required and an accompanying ValidationOptionAttribute. So I decided to take a look at that first.

I quickly found out that the ValidationOptionAttribute only works when used with instance dependency properties in Workflow Foundation. Workflow Foundation distinguishes between meta dependency properties, which are immutable at runtime and instance dependency properties which obviously can be changed at runtime. So this didn't work for us, because most of the properties we wanted to put some validation on were supposed to be bindable (for instance the To property on our SendMail activity).

So I had a look using .NET Reflector (an excellent tool by the way) to see what other activities we're doing. However I couldn't really find any activity that had both bindable properties as well as required properties. I quick search on Google didn't turn up much either. But I thought it must be possible so I had a go at implementing a validator using the pattern described earlier.

The problem with dependency properties that are bound at runtime is obviously that you can't know at compile time whether a value is available. So we decided to make it so that for a bindable property it must either have a specific value set explicitly, or a binding must be set (which implies that at runtime things might still go wrong, but that's another issue). Implementing the code for the validator wasn't especially difficult, but I still think it's a bit too much code for something as simple as this, but I'll let you decide. Here is the code for the validator:

internal class BindablePropertyDemoActivityValidator : ActivityValidator
{
public override ValidationErrorCollection ValidateProperties(ValidationManager manager, object obj)
{
    ValidationErrorCollection err = new ValidationErrorCollection();
    err.AddRange(base.ValidateProperties(manager, obj));

    BindablePropertyDemoActivity activity = obj as BindablePropertyDemoActivity;
    if (activity == null)
    {
      throw new ArgumentException("BindablePropertyDemoActivityValidator not applied to BindablePropertyDemoActivity", "obj");
    }

    if (activity.Parent != null)
    {
       if (String.IsNullOrEmpty(activity.Bindable) && !activity.IsBindingSet(BindablePropertyDemoActivity.BindableProperty))
       {
          err.Add(new ValidationError("BindableProperty is required", 7000, false, "BindableProperty"));
       }
    }

    return err;
}
}

This will make sure that the Bindable property of my sample BindablePropertyDemoActivity is either set to some static text or bound to some other property somewhere else in the workflow. If it's not, a red exclamation mark will be shown next to the activity and a red dot will also appear in the property editing window. In fact, compiling is a no go as well, because the validators are also called when the workflow is being compiled. This is nice of course, but it also poses a little problem I haven't been able to figure out.

As you can see in the code I'm checking to see whether the Parent of the activity isn't equal to null. If I remove that if statement I get an error compiling the activity itself. Although this makes sense, because when the activity itself is compiled the property will not be set to anything, I don't see such a check when reflecting the code of existing activities. Maybe because they mostly seem to check metadata properties only, I'm not sure. This approach works, but it means that the activity cannot be used as a root activity. I guess that isn't a big deal, because the classes provided by workflow should probably be at the root anyway.

What I still don't like about this approach is the amount of code required to do this stuff. I was thinking about making something a bit more generic (like a base class for validators that can check these kind of required properties itself), but I need to have access to the actual DependencyProperty object, which makes it a bit harder to make it generic. I guess I'll have to think about it some more :).

Silverlight 1.0 RTM

Jonathan Mezach — Thu, 06 Sep 2007 18:13:17 GMT

As my colleague Willem mentioned, Microsoft has finally released Silverlight 1.0. I find it amazing how fast this technology has gone. A year ago nobody heard of Silverlight, and now there is a first release. Although I'm quite impressed by the features of Silverlight 1.0, I think that Silverlight 1.1 has far more potential. Having the power of C# inside the browser, that must be awesome. It is going to open up a lot of possibilities for a richer end-user experience, while a lot of the advantages of web applications are still there.

What makes the release of Silverlight even more interesting is the announcement of a cooperation between Microsoft and Novell to get Silverlight to Linux. This is apparently the first contribution by Microsoft to open source desktop software. And although there is still a long way to go, I think it will benefit both sides. As a Linux user myself, I've always been a bit split between both sides. Developing ASP.Net applications and running them on Linux is quite possible these days, thanks to the efforts of the Mono project. But I was always running into a brick wall trying to do this, because some of the nifty features from ASP.Net weren't available. Of course, Microsoft will only support the Mono project in getting Silverlight up and running (they already have an implementation, but it's not feature complete and I couldn't even get it to work on my machine), but if this cooperation goes well, maybe we'll see some more developments in this direction. And wouldn't being able to run your .NET applications anywhere, independent of the underlying platform, be great?

More on this WCF issue

Jonathan Mezach — Fri, 31 Aug 2007 09:30:00 GMT

This week I've been spending some more time on this WCF issue I've blogged about before. Unfortunately I still don't have a solution for the problem, and I doubt that I will ever find one. However, I did find some interesting things I thought I should blog about.

I found out that the bug isn't in the actual serialization process, but rather in the generation of the code that is supposed to do the actual serialization. WCF's DataContractSerializer generates code on the fly to serialize and deserialize objects according to a specified data contract. The generated code is then called as a sort of anonymous delegate to perform the actual serialization or deserialization.

In the code that gets generated, serializing a nullable field is done by calling a function on the internal XmlObjectSerializerWriteContext class in the System.Runtime.Serialization namespace. This function is called GetHasValue which takes a generic type parameter T and a parameter of type T?, or a Nullable<T>. The function is static and it simply calls the HasValue property on the provided argument and returns the result.

A reference to this method is acquired using reflection, simply by calling the GetMethod function on a typeof(XmlObjectSerializerWriterContext). Sometimes however, this doesn't work and it returns null. This null reference is then passed to the MakeGenericMethod function which goes all the way down to the CLR which then fails because it is trying to dereference a null pointer.

What I haven't been able to figure out so far is why getting a reference to this static GetHasValue function sometimes fails and sometimes doesn't. There doesn't seem to be any pattern in when it fails and when it doesn't. I thought it might have something to do with the worker process unloading DLL's when the application pool is being recycled, but it's difficult to reproduce this behavior.

Here is a stack trace of when the problem occurs:

mscorwks!MethodDesc::GetMethodTable+0x12
mscorwks!MethodDesc::StripMethodInstantiation+0x8
mscorwks!RuntimeMethodHandle::StripMethodInstantiation+0xbd
mscorlib_ni!System.RuntimeMethodHandle.StripMethodInstantiation()+0x6
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Equals(System.Object)+0x64
mscorlib_ni!System.Reflection.CerHashtable`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon[], System.__Canon[], Int32 ByRef, System.__Canon, System.__Canon)+0x9e
mscorlib_ni!System.Reflection.CerHashtable`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Preallocate(Int32)+0x10e
mscorlib_ni!System.RuntimeType+RuntimeTypeCache.GetGenericMethodInfo(System.RuntimeMethodHandle)+0xef
mscorlib_ni!System.RuntimeType.GetMethodBase(System.RuntimeTypeHandle, System.RuntimeMethodHandle)+0x2a3490
mscorlib_ni!System.Reflection.RuntimeMethodInfo.MakeGenericMethod(System.Type[])+0x156
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlFormatWriterGenerator.UnwrapNullableObject(System.Reflection.Emit.LocalBuilder)+0x1cb
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlFormatWriterGenerator.WriteValue(System.Reflection.Emit.LocalBuilder, Boolean)+0x6f287
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlFormatWriterGenerator.WriteMembers(System.Runtime.Serialization.ClassDataContract, System.Reflection.Emit.LocalBuilder, System.Runtime.Serialization.ClassDataContract)+0x28d
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlFormatWriterGenerator.WriteClass(System.Runtime.Serialization.ClassDataContract)+0x6f27b
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlFormatWriterGenerator.GenerateClassWriter(System.Runtime.Serialization.ClassDataContract)+0x67
System_Runtime_Serialization_ni!System.Runtime.Serialization.ClassDataContract.get_XmlFormatWriterDelegate()+0x50
System_Runtime_Serialization_ni!System.Runtime.Serialization.ClassDataContract.WriteXmlValue(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.XmlObjectSerializerWriteContext)+0x17
System_Runtime_Serialization_ni!System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(System.Runtime.Serialization.DataContract, System.Runtime.Serialization.XmlWriterDelegator, System.Object)+0x2a
System_Runtime_Serialization_ni!System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(System.Runtime.Serialization.XmlWriterDelegator, System.Object)+0x8a
System_Runtime_Serialization_ni!System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(System.Runtime.Serialization.XmlWriterDelegator, System.Object)+0x1f

Possible WCF Serialization Issue?

Jonathan Mezach — Wed, 15 Aug 2007 07:34:00 GMT

As I was writing in my previous post there is something interesting going on in WCF. But before I go into the details, let me explain the situation.

There are 3 services in our project, one business service, one process service and one front end. We use WCF to communicate between these services. We are using basicHttpBinding and we have a central set of XSD schema's from which we generate code using the svcutil tool provided with .NET Framework 3.0. This has been working pretty good for us, but for a while now we are experiencing random exceptions when WCF calls are being made.

These exceptions are of type System.ExecutionEngineException. According to the documentation these exceptions are thrown whenever the .NET Runtime suffers from some internal error. However, the exception itself doesn't provide any meaningful information about what caused the exception to occur. In fact, we haven't been able to get any useful information from either the EventLog or our own logging framework. There are some messages in the EventLog but they only indicate that there was a fatal execution engine error and that the faulting module is mscorwks.dll.

Because the .NET Runtime itself is crashing, the application pool that is serving our service (we are hosting inside IIS) gets shutdown. If the exception occurs in the application pool serving our front end the end user is presented with a login dialog. Obviously this is unacceptable.

Unfortunately the exception occurs on pretty much all the machines we are using, from development to production. We haven't been able to reproduce the error consistently and enabling all types of logging and tracing that are available (including WCF's extensive logging and tracing options) haven't been able to provide us with a cause for this exception to occur. So after trying to establish the cause of this exception by using logging and tracing for a couple of weeks, we tried using the debugging tools provides by Microsoft to debug the ASP.Net worker processes.

What we have been able to find out so far that an exception occurs inside the service. We have written an extension for WCF which allows us to have exceptions that occur in the service to be automatically translated to either a FunctionalException or a TechnicalException, so these should be handled gracefully. Both FunctionalException and TechnicalExceptions are serialized and returned to the caller, but according to the stack trace we have been able to retrieve something goes wrong while serializing the exception.

We are still not sure what is causing this, but something goes very wrong. We are currently trying to create a full dump, instead of a minidump so that we hopefully can see what data is being passed around. If we found out anything new, I will definitely blog about it again.

A short introduction

Jonathan Mezach — Wed, 15 Aug 2007 06:29:00 GMT

As I'm new here on the Info Support blogs I thought it would be a good idea to introduce myself a bit.

My name is Jonathan Mezach and I have been officially working here at Info Support since the 1st of august 2006, so I'm here now for a little more than a year. But I already started working at Info Support about a half year earlier in order to finish my degree in Software Engineering.

These days I'm doing .NET Development and currently I'm working on a moderate sized project using .NET 3.0, or at least Windows Communication Foundation and Windows Workflow Foundation. We have found something interesting about WCF during this project which I will blog about soon.