Craig Brimhall, Harvard-trained L&D Instructional Designer
Experiments are the fundamental building block of science. In the past, experiments were usually conducted by scientists at universities, but today many businesses have recognized the value of experimentation and have started to recruit university-trained scientists to run experiments with their brands, products, and user experiences.
A well-designed experiment uncovers useful information about the world: illuminating facts, patterns, and causal relationships.
For example, Booking.com runs daily experiments on its website to learn how differing the font, the placement of pictures, and the choice of words influences its customers’ behavior and experience.
In organizations, the ability to design and run effective experiments will help to “future-proof” them against unpredictable shocks.
Through experiments, senior leaders and managers can mitigate risks and capitalize on opportunities by testing new ideas and learning which methods and concepts contribute to the best outcomes. As the chief product officer for Booking.com said of experimentation, “You have to embrace it or die.”
_________________________________________________
In science, experiments have been used to figure out human genetics, to discover vaccines for debilitating diseases, to understand how humans make decisions, and to test strategies for reducing climate change. In organizations, experiments have been used to understand the traits of successful leaders, to reduce workplace discrimination, to increase employee motivation, and to improve brand recognition.
A well-designed and well-executed experiment generates valuable data that can determine how one change, however small, influences an important outcome, like performance.
So what makes an experiment well-designed and well-executed? Although there are countless different forms of experiments, effective experimentation has three essential characteristics: a clear target, a precise plan, and a quantifiable measurement.
An experiment run by Disney is an excellent case study to understand the three characteristics of experimentation. For over sixty years, Disney has demonstrated a strong commitment to sustainability, manifest in its donations to wildlife preservation, the nature documentaries it produces, and its corporation-wide efforts to reduce emissions and conserve water. One of the largest demands for water within the corporation comes from guests staying at Disney resorts, in part due to the water needed to wash towels and bedsheets every day—accounting for 16 percent of their hotels’ overall water use. Not only would a reduction in this water usage be good for the environment, it would also be good for Disney’s bottom line. But rather than engaging a focus group, bringing in external sustainability consultants, or looking to other companies for ideas in reducing this usage, Disney decided to run an experiment. What would happen if the company gave hotel guests a pin in return for signing a commitment-to-conservation pledge? Would this influence a significant number of guests to re-use their towels instead of requesting a new towel each day?
1. A Clear Target
What is the target of Disney’s experiment? If you said “to conserve water,” you are right, broadly. If you said “to persuade guests to reuse towels,” you are also right. Conserving water is the ultimate goal of the experiment, but persuading guests to reuse towels is the target—the specific outcome or behavior that the experiment is addressing. A goal is often a broad aspiration like “improving environmentally friendly practices,” whereas a target is specific and measurable, like the number of guests who reuse their towel.
2. A Precise Plan
An experiment determines how a specific change impacts a particular target. Being able to determine an effect accurately requires a precise plan. A good experiment changes only one factor at a time. This is critical because any observed difference in outcomes can then be attributed to this one change. Disney’s plan was to give some guests—but not all—a Friend of the Earth pin if they signed a commitment to the environment and pledged to reuse their towels. The company could then compare the rates of towel reuse between guests who were offered a pin and those who were not.
3. Quantifiable Measurements
To assess the effectiveness of an experiment accurately, there must be a quantifiable outcome to measure. Determining how to perform the measurement can be tricky. There are multiple ways to measure towel reuse: (1) the researchers could count the total number of towels that appeared to have been used and then hung up; (2) the researchers could subtract the number of towels replaced from the total number of towels in the guest rooms; or (3) the guests could be asked to self-report. Disney chose to have its researchers visit the guests’ rooms when the guests were out and count the number of towels that appeared to have been used and then re-hung.
The results of the experiment showed a twenty-five percent increase in the number of towels reused by Disney’s guests who had had the opportunity to receive a pin. Thanks to the experiment, Disney learned how to encourage guests to reuse their towels and save water, thus contributing to the larger organizational goal of environmental sustainability.
_________________________________________________
There are several ways to mitigate the risks of experiments while still reaping their benefits:
A. Follow the three steps to creating an effective experiment explained earlier: designate a clear target, create a precise plan, and ensure the possibility of quantifiable measurements. It will be much easier to identify why the experiment failed and what can be learned from it if these three elements are in place.
B. Adhere to the principle of starting small and simple. Small and simple experiments keep costs down and reduce the impact of failed experiments on your organization.
C. Involve the right people. When considering a team for running your experiment, make sure to include both people with high power and people with high interest. This way the team will have the authority as well as the interest to carry out the experiment as best they can, capture the results effectively, and disseminate what has been learned throughout the organization. Those in positions of power can also provide the tangible and moral support necessary to running the experiment and sharing the findings.
D. Create formal policies that define the parameters of the experiment and who will be involved. Not every problem is suited to resolution through experimentation. Some problems require experiments that would be too expensive; others are too delicate or politically charged to be amenable to experiments. The best problems to solve through experiments are those in which you suspect a particular change may improve a certain outcome; on top of that, you must have the time to test different ideas and the minimal costs needed to run the experiments. Once you’ve established an experiment’s feasibility, formal policies will help it run smoothly, without causing friction or confusion in your organization. They may also help to minimize the productivity loss resulting from the experiment.
This excerpt is from Craig Brimhall’s chapter, “The Power of Experiments to Meet the Future Head On,” from The Center for Asia Leadership’s Rethinking Asia 7 – The Future of Work: How To Prepare For It. To learn more, please visit http://asia-leadership-2023.local/publication/.
