Stop Doing Experiments and Start Making Bets
Many teams have embraced the experiment mindset, which teaches that you can learn more from simply trying out an idea and seeing the results than you would ever learn from intense upfront analysis.
While this mindset can help your team make rapid gains, there’s also a dark side to this approach. Although experimentation can yield faster and more in-depth learning than traditional analysis, many teams forget that some initial lightweight analysis can also be useful for helping them choose those experiments that are the most likely to succeed. As a result, these teams embark upon poorly planned and risky experiments without first considering what they stand to gain...or lose.
The root of the problem lies in the word “experiment”. By their nature, experiments can be very open ended. This can make them a bad model for change since the lack of a defined end state can lead to a lack of clarity on how to proceed, or even a lack of understanding of what the team stands to gain from the experiment.
Playing the Game
A better model is a bet. Imagine playing a game of poker but with no chips. Without a need to commit chips then each hand becomes nothing more than a simple experiment with no potential upside or downside. This means that you could play forever and still have no real investment in the game.
But adding chips to the hand changes the game. When you’re working with a finite amount of chips you start to become very aware of the real possibility of gain and the real possibility of loss that hides in each hand. This forces you to weigh your options much more carefully.
Watch a great poker player play the game and you may be surprised at what you see. Great poker players don’t play every hand...in fact, many even sit out more hands than they actually play.
Why? Because great poker players know that they have a finite amount of chips and, if they want to play the game long enough for that big pay out, then they need to protect the chips they have. This means that they evaluate each hand at the outset and decide if it’s a hand they stand a chance at winning. If they’re not likely to win, then they simply bow out and wait for a better opportunity. This is because they know that every hand has a cost, and the more bad hands that they pass on, the more likely they’ll still be in the game when those truly great hands come along.
Betting on Change
So should you abandon experiments altogether? Not so fast. A true embracement of an experiment mindset is one of the hallmarks that separates the best teams from the mediocre ones. The difference, is that rather than simply diving headfirst into every opportunity, the best teams evaluate each of the possible experiments on their plates and choose to focus on those experiments that have the most potential for a positive upside, given the investment they’ll need to make.
For example, imagine that during the most recent retrospective your team complained that their Product Owner is not easily accessible throughout the sprint to provide feedback or clarification on items that are in progress. As a result, your team has repeatedly found themselves reviewing items at the Sprint Review that turned out to not be what the Product Owner had in mind. This means that they’ve had to invest additional time in the next sprint to rework those items, which has slowed the team down as a whole.
After a few minutes of brainstorming your team comes up with several experiments they could run which could help address the problem. For example, maybe the Product Owner could appoint a proxy Product Owner on the team that would be empowered to provide feedback and make decisions on the Product Owner’s behalf whenever he’s unavailable. Or, perhaps the Product Owner could make more of an effort to attend the team’s standup whenever possible so he’s available to provide feedback to the team as items are complete. Of, maybe he could even keep regular “office hours” in the team room so he’s more accessible to the team a few days per week.
Placing the Right Bet
Any of these options could be worth pursuing, and when each is considered as an experiment they all look equally promising. It’s only when we look at these options through the lens of making a bet do the tradeoffs between each become more clear.
The table below demonstrates a simple framework for evaluating different experiments through the lens of a bet.
| Bet | Potential Upside | Potential Downside | Odds of Success |
|---|---|---|---|
| Proxy Product Owner | Quick response to questions | Proxy may provide wrong responses leading to future rework
Proxy role becomes a distraction from that team member’s responsibilities |
Unlikely |
| Product Owner attends standups | Slightly more availability to team
Quicker response to questions, if asked during standup |
Greater impact on Product Owner’s time | Possible |
| Product Owner office hours | Tighter collaboration with team
Quicker response to questions Less impact on Product Owner’s time |
Different way of working for Product Owner | Likely |
When a team starts to look at experiments through the lens of a bet, not only does the potential for gain become clearer, but they also better understand the investment that they’ll need to make to carry out each experiment. This can help them better separate those experiments that are likely to fail from those that have a better chance of success so they’ll know where to focus their efforts.
Knowing the Cost
An experiment mindset can be a powerful tool for teams who desire to improve their way of working. But, while experiments may appear to be free, few truly are. Most experiments carry with them very real costs, but by evaluating the potential for loss or gain of each your team can focus on those experiments with the greatest likelihood for success...meaning that they’ll be around to play the game for a long time to come.