0 comments
No Category
Does your team ever struggle to decide which experiments it should do?
In a previous post, I explained why experimenting is essential for Scrum teams to remain effective. Teams can come up with many different experiments: for example, changing the type of beans in the coffee machine, working remotely or moving to a different office building. But how do we choose which experiments are the most worthwhile to try? In this post, I explain what makes one proposed experiment more appealing than another. This should make it easier to make a choice.
Judging Experiments
Experiments are often judged after they have been done, based on their results. If you experimented with a change that increased development speed by 50%, then you say it was a good experiment. If you experimented with a change that made your customers angry, then you say that it was a bad experiment. We cannot use this kind of reasoning when choosing between experiments, because we do not yet know what their effects will be. Instead, we have to predict the impact of the change.
Taking that into account, I use the following definition:
The appeal of an experiment is defined by its expected increase in the team’s effectiveness.
Thus, if we have to choose between experiments A and B, and we expect A to increase our effectiveness more than B, then we should choose A.
Two things stand out in this definition. Firstly, it talks about the expected increase – but expectations can be subjective. One person may expect a change to lead to great improvements, while somebody else may be pessimistic. Just as expectations of the improvements are subjective, so is the appeal of experiments. An experiment may be more appealing to one person than to another. The team needs some way of reaching consensus. To do so, the team members should explain to each other why an experiment is appealing to them or not. In other words, they explain their expectation of its impact on the team’s effectiveness.
Secondly, the definition talks about an increase in the team’s effectiveness – but how do we measure team effectiveness? In general terms, a team is more effective if it “delivers more value” per time unit. It varies what organizations see as value. For example, a commercial company may think of profit, customer satisfaction or carbon footprint, whereas a hospital may think of the number of healed patients or the increase in their patients’ quality of life. It is important that the team knows what its organization finds valuable. If they know that, they can try to optimize the value they deliver by doing experiments. If the team expects an experiment to lead to great media coverage, but their business does not care about media coverage, then this proposal is not appealing. In short, we measure a team’s effectiveness by the value it delivers, as defined by the team’s organization.
Putting Theory Into Practice
We now know theoretically what makes an experiment more appealing than another experiment. We must still, however, translate this theory to practically choosing between experiments!
Let’s consider an example. A hospital has tasked a team with developing an application that allows doctors to see their patients’ medical information. The team delivers value by improving this application. This team depends on the doctors to tell them what would improve the app. One team member proposes to bake a cake before every meeting with the doctors, so that more doctors will show up, so that they will know better how to improve the app, to increase their effectiveness. Another team member proposes to start working remotely one day of the week, which may lead to increased focus for the developers, which increases their effectiveness. The team agrees that they only want to experiment with one of these options for the coming sprint.
To choose between these two experiments, the team tries to estimate which proposal has a higher expected increase in the team’s effectiveness. The team members explain to each other how much they expect each option will increase their ability to improve the application. They discuss various aspects:
- If the experiment is a success, how much does the team gain? And for how long will these benefits last? For example, the benefits of baking cakes may diminish over time if the doctors get used to them.
- If the experiment is a failure, how much damage can it do? For example, if working remotely turns out to reduce focus instead of increase it, the team may have wasted some effort by doing the experiment.
- How likely is the experiment to succeed or fail? An experiment that is likely to fail may still be worthwhile to try if it could lead to great benefits if it succeeds. And an experiment that is likely to succeed may not be worthwhile to try if it would only lead to very small gains.
An experiment’s appeal is a combination of these aspects. It is not the expected result of the most likely outcome (which would be bad if the experiment is most likely to fail). Instead, it is weighted by the “goodness” of a success, the “badness” of a failure and the expected chances of success and failure. This is illustrated in the figure below. Even though the most likely outcome is negative, the appeal of the experiment can be positive because a failure would not be very bad and a success would be very good. Drawing such graphs can help to explain appeal, reach consensus and choose better experiments.
Choose Your Experiments
Choosing good experiments is important for Scrum teams to remain effective. Each team member may have a different idea about which choice is the most promising. To make a choice, the team needs to reach consensus. To do so, it helps to talk about the reasons why certain experiments are appealing. They are appealing if they are expected to increase the team’s effectiveness. We do not know the results of proposed experiments beforehand, but choosing the most appealing ones will make the team the most effective.