A Deep Dive into Strategic Experimentation
Understand how to conduct effective experiments that align with your company's strategy
First thing, first
Research and Experimentation are not the same. Research helps to understand market needs, user preferences, and industry trends. And, Experimentation is focused on validating hypotheses by testing specific solutions.
In the Product Discovery Process, a strategic combination of research and validation methods is useful to mitigate risks and reduce the uncertains regarding product development. Each one has different purposes, approaches, and findings and can be executed with various techniques.
When to run experiments?
Running experiments allows you to validate new features, optimize conversions, and mitigate risks when expanding a product into a new market. Experiments can be used when facing a significant problem that impacts company outcomes and you have the minimal structure required to run the experiment.
Experiment process
According to Reforge, the experiment process consists of four steps. Begin by mapping out Strategic Opportunities, then move on to Developing Solutions. Next, engage in Testing and Learning, and finally, in the last step, proceed with the Productization of the solution.
1. Identify Strategic Opportunities
Start by understanding the company strategy, mission, goals, and the current state of the product, problems, and opportunities. The composition of a strategic opportunity is experiment strategy, user problem, and business outcome.
1.1 Strategy: To guide experiments, it's important to understand what are the organization's goals and how the product plays a role in achieving those goals. This is possible by understanding the product growth model, or in other words, how your company is still alive or keeping a sustainable business.
This means figuring out:
How does the product attract new users?
How does the product keep users engaged?
How does the product make money?
These definitions will be valuable when prioritizing the experiment and the main expected outcome. It probably will be around acquisition, engagement, retention, or monetization.
1.2 User Problem: The customer problem represents a barrier to product growth. Solving this challenge will unlock value for both, users and the business.
Try to answer these questions to define a better problem statement:
What is the possible root cause of the problem?
Which clusters of users are most impacted?
Which user’s behavior or use case is impacted by the problem?
Which strategic outcome this problem is impacting?
Attention points:
Seasonality: When analyzing data to comprehend user problems and behaviors, consider the influence of timing, holidays, vacations, and special events that impact the product. For example, on an e-commerce platform, you might notice an increase in the demand for summer clothing during the summer season and an uptick in searches for winter clothes during colder months.
Unqualified users: Analyse the segments that are having these problems and identify if they have the necessary characteristics or requirements to be part of the specific product target user base. For instance, users who are too engaged or are already dormant can have a lot of influence on how the user problem will be shaped.
1.3. Business Outcome
Metrics are important for tracking experiment results and showing if the expected impact was achieved. Imagine a test on a website's checkout to make buying things online easier. Connect the results to bigger business goals, proving how a better checkout helps the whole business. If increasing the Net Revenue is a big goal for the company, show how the experiment increased the conversion rate and how it can be translated to revenue. This approach helps understand the relation between the product and the business and also can facilitate communication with stakeholders, partners, and C-level. Check this other example below:
2. Develop Solutions
In this step, focus on generating solutions to address specific problems. Teams normally use benchmarking for inspiration and brainstorming ideas as a starting point for sketching potential solutions. Even while planning an experiment, consider methods like prototyping and usability tests to mitigate risks before rolling them out to a bigger audience. The main goal is to discover and implement the most effective solutions, balancing effort vs impact to validate the hypothesis.
These solutions can have different levels of value and innovation, having a low, medium, or high classification.
Some examples focused on an E-commerce App could be:
Simplify: Remove an obligatory field in account creation.
Enhance: Improve the search functionality by adding filters.
Reorder: Reorder the categories on the homepage based on the popular items.
Restructure: Restructure the app's menu to categorize products more intuitively, making it easier for users to find what they're looking for.
Addition: Add earn and burn mechanisms, by creating a points system for every purchase to incentivize customer retention.
Reivent: Reinvent the shopping experience by integrating augmented reality for virtual product try-ons before purchase.
3. Test & Learn
Once you have solutions, you start to test the solution. To test the prioritized solution, you can use these types depending on the specific objective you have.
A/A Testing - Test comparing two identical versions (A and A) of a webpage or product to make sure the testing process is working correctly. This helps avoid mistakes and ensures consistency. It's an approach used to validate technical structure, like test platform consistency and reliability.
A/B Testing - Test to compare two versions (A and B), variant and control, of a webpage or product to see which one works better for users. It helps improve things like headlines, images or calls to action based on real user behavior. Use to compare different versions to find which version has the most effective results.
Multivariate Testing - Test to validate multiple variations of different elements on a webpage simultaneously compared to the control group. It's useful to understand how different changes work together and find the best combination for the best performance. Use to analyze the impact of changes of multiple elements in the user behavior.
4 . Productize
Success in experimenting means putting good ideas that have presented good results, to be implemented. It's important to think about:
Impact - What should be the impact on the main outcomes of productizing the solution?
Cost - How much productizing the solution will cost?
Strategy alignment - How well does it fit with company strategic plans?
Scalability - How easy is it to scale the solution for a bigger audience?
After having this clarification, you can start by expanding the solution to a part of the base, and making it available progressively.
Key outcomes
Mitigate risks: Embrace a strategic combination of research and experiments to mitigate risks and uncertainties.
Understand the product growth model: Try to start by understanding what are the strategy and main levers of the organization.
Focus on real user problems: Before creating and testing solutions, identify and prioritize what has more impact on the product growth model.
Start experimenting: Even if you are not so confident, and feel that you don't have all things in place, start doing and then evolve the process.