Why Experimentation Is So Important?
According to Richard Feynman, very famous American physicist, his simple statement holds the key:
“If your guess disagrees with experiment, it is wrong. It doesn’t matter how beautiful your guess is, how smart you are, who made the guess, or what their name is . . . it is wrong.” — Richard Feynman
If your hypothesis disagrees with experiment, it is wrong! Experiments are not only real-life and material experiments. Galileo’s and Einstein’s thought experiments are also famous experiments. But experiment goes beyond science. Writing a novel is about inventing a character and trying to explore many possibilities to see how we can walk him in a fictional reality. We trust a certain hypothesis in a determined situation where we can imagine but not realize. It’s like a laboratory of the mind. A lab in your head. Isn’t it pretty cool ?
Let’s not go too far, let’s go back to the real world. Richard Feynman has developed the theory of quantum electrodynamics. He and two others shared the 1965 Nobel prize in physics. This theory describes how light — electromagnetic waves — propagate through space at the quantum level. He described the scientific method using the following three steps :
Hypothesis → Compute consequences → Compare to experiment
What if an experiment fails to confirm your hypothesis? Assess all stages of the experiment, analyze the outcomes and make small changes. Alter the experiment and revise the hypothesis. If several different experiments all reveal the hypothesis has not been confirmed, a revision of the hypothesis is needed. After several reproducible experiments showing the hypothesis does not work, then it might be time to reject the hypothesis and replace it with a more viable one.
In scientific experiments, outcomes are data! It is indeed essential here to differentiate between conclusions and data. In science, conclusions are weak, compared to actual data. In the future, the conclusion can turn out to be false, or incomplete. Your data can be re-analyzed and show this in fact. That is why in real world data is so important.
“In God we trust. Everybody else brings data to the table.” — Narayana Murthy
This very famous quote of Narayana Murthy, and well-recognized cliché, seems to mean. In fact, data removes subjectivity during decision making and provides basis for a fair mental model built on data driven decisions. Mental model is the way people think, understand and act in the world. Everyone has a mental model, their own way of looking at the world.
It’s helpful to consider multi levels of experiments to challenge this mental model. Let’s explore 3 levels of multilevel experimentation: experiments at personal level, at team level and at enterprise level.
At Personal level
Recognizing the importance of experimentation is probably the first step. Experiments are a crucial source of the data necessary for the activation of knowledge. Keeping an open mind with experimentation helps you to learn and grow, strengthens your belief in yourself.
Lack of data has long been a major barrier to experimentation. But we are in the “Age of data”. Today, accessing to data is very cheap. With quantified self apps you can track physical health, food, time, and money with applications like MyFitnessPal , LifeSum, ActionDash, RescueTime, YNAB, Toggl (even meditation like Calm). Many of these apps have free features. It’s very easy to collect a lot of data without any constraints. This frictionless-entry assumption signifies a minimal cost to collect, analyze, centralize and aggregate data. Using these data as metrics, you can target outcomes and experiment in many areas of your life (productivity, health, creativity, self-sufficiency…). You can develop yourself with a permanent tracking using continuous improvement. This individual continuous improvement enables learning behaviors. Individual learning behaviors develop yourself. It gives you the opportunity to become a better person whatever the goal you try to achieve.
In addition, be careful with “vanity metrics”, indicators which appear to have a very particular appeal, but which don’t allow as much to understand your performance. These indicators are generally used to provide an improved impression, but don’t give rise to any measurement and are not associated with any controllable or reproducible factor. Vanity metrics are fickle, falsely encouraging and very attractive. It’s the feeling to grow up without impact on your development.
At Team Level
Working at the team level, traditional micro-management instills a mindset that enables efficient execution and actually inhibits the team’s ability to learn and innovate. The narrow focus on getting things done inhibits the experimentation that is essential to success in an unpredictable business environment and rapidly-evolving world.
In high-performance teams (HPTs) that produce superior results with high levels of collaboration and innovation, the main purpose is to experiment using rational decision making methods and generate new possibilities. Experimentation means expecting not to be right the first time. It is a way of doing that involves learning from the results of action, mostly based on data. In team dynamics, experimentation behavior involves evaluating and testing the impact of one’s ideas and actions with respect to what other team members are thinking. Experimentation is an essential aspect of a HPT because of the uncertainty and it’s also a crucial part of learning.
For a technical team, experimenting and learning rapidly is part of technical excellence. Active experimentation is one of the main activities of collective learning. Permanent experimentation and debate lead to possibilities that are attempted, rejected, transformed, adapted, and refined. Numerous tradeoffs can be evaluated, and teams can resist easy compromises, pushing instead for elegant solutions to tough problems they confronted.
For instance, if there is one thing that Agile (and Scrum) teams can experiment is the sprint length. It’s easy and even essential.
Using a data-driven approach, teams are able to identify trends over time which can inform effective practices, helps to become aware of issues, and illuminate possible innovations or solutions. Experimentation is critical to the success of a team.
At Enterprise Level
The equivalent to a scientific experiment in innovation is a cycle around the validated learning loop, popularized by Eric Ries in this book The Lean Startup.
A set of ideas or hypotheses are used to create experiments in the build stage. It begins in the “build” stage, a set of ideas or guesses are used to create some prototypes for the purpose of testing the idea. This prototype is used to “measure” customer responses using a combination of qualitative and quantitative techniques for gathering data. This data is then used to derive specific “learning” that either confirms or rejects our hypothesis. Fail fast, fail cheap ! That in turn drives the next set of ideas which inform the next set of actions. Three key attributes for running effective experiments are speed, customer learning, and focus.
Companies introduce new products and services that they believe are an improvement over what they offered before. Experimentation on a small scale allows them to foresee the outcome on a larger scale. Data driven approach is very important to limit risks, and to find a path on your business objectives, and provides a strong foundation for a long lasting enterprise. Setting SMART goals are essential to map a direction. If you have vague or ambiguous goals, you won’t end up where you want to be.
Many large enterprises have used experiments, and tech giants such as Google, Netflix, Tesla and SpaceX run a lot of experiments each year. Michael Luca and Max Bazerman, in “Experiments in the Tech Sector” part of their book “The power of Experiments”, explain how Google can run thousands of experiments per year.
« Google now runs experiments at an extraordinary scale — more than 10,000 per year, about half of them related to Google’s advertising products and the other half related to its search engine. The assumption within the company is that the results of these experiments will inform managerial decisions in a variety of contexts. » — Michael Luca, Max Bazerman
Your intuition is important and making decisions without considering intuition are disadvantageous. But the reverse is worse. The confidence of intuition and guess must be moderated. Overconfidence in the intuition and the ability to guess the effect of an intervention is in fact a big problem. Testing and balancing intuitions is a good way to develop intuition as a skill.
Overconfidence syndrome can lead managers and executives to a miscalibration of their goals and objectives, and make decisions on their gut rather than to experiment to determine what the best course of action would be. Google, Facebook, and Amazon, for instance, vary what individual customers see, simplifying the randomization process and implementation of experiments. The major tech platforms now all have techniques (like A/B testing) techniques to randomize and to easily show some users one design of a webpage and others different one, and then collect data about user behavior across the different designs. Of course, these big companies have organized strategy in favor of data quality and Big Data, hence they do not experience the negative impacts of inaccurate or corrupt data.
You think probably collecting data and managing the quality data can seem like an overwhelming task in your company. At Continuous, we helped many SMBs and startups overcome in adopting data quality processes and capturing value from analytics at team and enterprise levels. In many business cases, we guided clients to ramp up their ability to explore and interrogate data so that they can answer their most important business questions and validate their hypothesis. We start with a workshop to meet your exact training and business needs. It is meant as a starting point only to help our clients to build successfully a fast data architecture.
As we have seen, you are probably not a big tech company however you need never stop questioning your intuitions and hypothesis.
How much have you improved in a particular area at personal level ? At team level? How many experiments have your company made this year? How do you know if you are getting better compared to last month? Last year? Are actions actually aligned with priorities?
Do you have some trouble answering these questions? Share with me through comments. If you want to know more about our business approach, contact us.
Visit Continuous.lu and learn more about cloud data infrastructures and managing data quality!
