When using MS Release Management, I found that during a Release, deploying a Powershell/DSC component always takes at least one minute to complete, even if the actual work within that step (copying the files and executing the Powershell script) takes less than a second. This effectively means that with every component that we added to our release template, this increased the duration of the release with another minute.
Especially if you want to minimize application downtime, this seems like an overly long time to wait for nothing.
Logging to the rescue again
So lets see if we can find out why this is happening. MS Release Management consists of several parts, and the one performing the actual Release Management process appears to be the Release Management Monitor, which is run as a Windows Service. It has its own ReleaseManagementMonitor.exe.config
file (located in MSRM’s bin
folder), and using it, we can turn on logging by replacing the <system.diagnostics>
section with the following fragment:
<system.diagnostics> <trace autoflush="true" indentsize="4" /> <sources> <source name="DTLHosterTracing" switchValue="Information"> <listeners> <remove name="Default"/> <add name="DtlListener"/> </listeners> </source> </sources> <sharedListeners> <add name="DtlListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:LogsReleaseManagementMonitorDtl.log" traceOutputOptions="None"> <filter type="System.Diagnostics.EventTypeFilter" initializeData="All" /> </add> </sharedListeners> </system.diagnostics>
In order to have these settings take effect, the service should be restarted. In Powershell:
Restart-Service "Release Management Monitor"
If you deploy a release with these settings, you’ll find that as soon as the Deploy stage starts a new logfile is created with some very useful information:
Adjusting the Deployment Heartbeat
As you can see, there appears to be a configuration setting called DeploymentHeartbeatValidateIntervalSec
that defaults to 30 seconds. During the deployment of a component, this heartbeat period always appears at least twice (see screenshot). Why, I don’t know, but the actual deployment of the component always seems happen on the second heartbeat – this can be seen by the output from “Hello.txt” (see inset), which is what my Component produced.
Knowing this, it’s a pretty safe bet that we can add an appSetting that overrides the duration of this heartbeat. And indeed, if you add the following to the ReleaseManagementMonitor.exe.config
file (don’t forget to restart the service)…
<appSettings> <!-- ... --> <!-- Determines how often the remote windows service is polled to see if it has finished deploying; defaults to 30 seconds. --> <add key="DeploymentHeartbeatValidateIntervalSec" value="5" /> </appSettings>
… you’ll find that each component now only takes a minimum of 10 seconds (plus some overhead because of “real” work) to deploy:
Note that I wouldn’t recommend going lower than 5 seconds (or even 10 seconds), because as you can also tell from the logs, on each heartbeat a new PSRemoting connection is set up, which actually spikes the CPU – and you don’t want to hammer every machine during the deployment.
Adjusting the component deployment interval
With the heartbeat adjusted, you might also notice that every component is being deployed on a 15-second boundary:
Luckily, this interval can be reduced too by using the DeploymentEventFetcherInterval
appsetting:
<!-- Determines the rate at which is checked if the next Component can be deployed (i.e. how fast new rows are added to the "Deployment Log" window). Default is 15 seconds. --> <add key="DeploymentEventFetcherInterval" value="5" />
Hope this helps!
Note: A colleague of mine told me that after restarting the Release Management Monitor service, every Release Management client started after that showed the message “This license has expired” and refused to go further, even though they’re using the fully licensed version.
I haven’t experienced this myself, but if this happens to you, recycling the Release Management AppPool in IIS (or simply doing a full “iisreset”) should make MS Release Management recognize again that it’s running the full version.
I don’t believe it’s directly related to the modifications described above but rather some sort of general hiccup that sometimes happens.
2 comments
First of all, thanks for this very helpful blog. It has been rather difficult to find solutions to some of the issues inherent to Release Management. This particular problem has caused a lot of frustration as the Release Template we use consists of 26 components per environment. Each component took at least 1:45 to complete, with some of course taking a bit longer depending on their actual workload.
Using your suggested solution (setting both DeploymentHeartbeatValidateIntervalSec and DeploymentEventFetcherInterval to 5 seconds), we brought the minimal required time down to approximately 0:55. While this is obviously a vast improvement, we were still a little disappointed after seeing your screenshot of a component taking a mere 0:17 to complete. Would you say this low time is purely because it wasn’t a realistic example and any real world scenario (with for example a Configuration script included) is guaranteed to take much longer? What sort of times are you witnessing with your own actual releases? We’re mainly curious as to whether we should continue our quest for further reduction in release time or give up and accept. Any other tips you might have are welcome as well of course!
Kind regards,
Bart
Bart l'Amie
Hi Bart,
Nice to hear that this information was helpful to you. Although the 0:17 in the screenshot is taken from a demo setup, a duration of around 20 seconds should definitely be possible if you’re not copying large files or doing too much processing in your script.
I would recommend creating a simple Release that just copies and runs a simple “Hello world” Powershell script and see how long that takes, to rule out that file size or actual processing time are the delaying factors.
Next, you could try to remove MSRM from the equation by setting up your own PSRemoting connection from the MSRM server to the target machine – see Secrets of Powershell Remoting for an excellent explanation. if this takes about as long as via MSRM, MSRM is not the problem, but maybe something like resolving host names or setting up the WINRM connection.
If your own PSRemoting connection does work much faster, it could be that making the connection back from the target to the MSRM machine (to copy the component files from the file share) is what’s causing the delay, so you may also want to try to isolate that part. Just a couple of ideas…
Léon Bouquiet