Insights on the ground

A/B testing is a programme of work that is run by the Runway team. Broadly speaking, this is what is delivered to the client:

1. Test Results: Data on performance metrics, such as conversion rates, click-through rates, bounce rates, and other key indicators. Identify the better-performing variation.

2. Insights and Recommendations: Summarize findings, insights, user behavior analysis, and improvement recommendations.

3. Documentation of Changes: Detailed documentation of recommended/implemented website changes, including design mockups, code changes, content updates, and implementation instructions.

4. Visual Comparisons: Visual representations of tested variations to understand design and content differences.

5. A/B Test Plan: Initial plan outlining objectives, hypotheses, target audience, test duration, and important details.

6. User Feedback and Comments: Qualitative data collected during the A/B test to gain insights into the user experience.

7. Statistical Significance Analysis: Report on the significance of results to determine meaningful differences.

A/B testing is a cost-effective experimentation method for comparing different versions of web pages with real users. It helps determine which design, content, or layout changes improve performance and achieve specific goals. Statistically proving the effectiveness of a winning page, provides confidence for rollout.

An A/B testing programme is a powerful way to improve your websites, mobile apps, or marketing campaigns. It compares different versions (A and B) of an element or feature to determine which one performs better in achieving goals. Here's an overview of our A/B testing program:

1. Goal Definition: The program starts by setting clear objectives. Organisations identify the key performance indicators (KPIs) they want to improve, e.g. conversion rates, lead generation, or engagement metrics.

2. Hypothesis Formation: Based on the goals, educated guesses are made about which changes to test. For example, if the goal is to increase email sign-ups, a hypothesis could be that changing the color of the call-to-action button will improve conversion rates.

3. Test Design: The A/B testing team creates different versions (A and B) of the element or feature to be tested. Variation A is typically the existing version, while variation B is the modified one. Care is taken to ensure that the test is conducted with a representative sample of the target audience.

4. Implementation: The versions are implemented on the live website, app, or in marketing materials. Users are randomly assigned to one of the versions, and their interactions are tracked and recorded.

5. Data Collection: Data is collected on user interactions and conversions for both versions. The program runs for a specified duration, during which the data accumulates.

6. Analysis: Statistical analysis is performed to determine the significance of the differences observed between versions. This analysis helps decide if the improvements are statistically valid.

7. Decision and Implementation: Once the test reaches statistical significance, a decision is made to adopt the better-performing version. Changes can be implemented on the live website or in the marketing campaign.

8. Iterative Process: A/B testing is an ongoing process. Organizations continue to test and refine elements based on data and user behavior.

9. Documentation and Reporting: Results, insights, and recommendations are documented and reported to stakeholders. This information informs future testing and helps understand user preferences and behaviors.

10. Scaling: Successful changes can be applied across the entire digital platform, and new hypotheses are continually generated for further testing.

A research report that presents prominent themes along with customer 'pains and gains'.

Depth interviews address the limitation of surface-level data by delving deeper into individuals' thoughts, emotions, and experiences to gain a richer and more comprehensive understanding of complex issues or phenomena.

Qualitative depth interviews are a research method where open-ended questions are used to explore participants' experiences, opinions, and emotions in detail. This process helps gain profound insights into consumer behaviour, preferences, and attitudes.

During these interviews, participants are encouraged to express themselves freely, allowing researchers to uncover nuanced insights that quantitative surveys often miss. The flexible nature of depth interviews allows for follow-up questions, to probe deeper into participants' responses.

These interviews are valuable when trying to unpick complex issues, By delving into the intricacies of human thought and emotion, businesses can refine their strategies, enhance product offerings, and tailor marketing approaches more effectively.

A test report that shows which navigation structure(s) tested performs the best by reporting the time it took users to find information and the number of users who successfully completed the task. Researchers also collect overall feedback on the structure.

Website tree testing helps evaluate how easy it is to navigate a website and find information. It identifies problems like confusing menus and hidden content. This improves the user experience by making sure users can quickly find what they need on the website.

The process starts by creating a simplified version of the website's structure, known as a "tree" or "site map." This tree shows the organisation of pages, sections, and content categories on the website.

Test participants are shown this tree and given a specific task, like "Find information about the company's returns policy."

During the test, participants use the tree to find the information they need on returns. They click on different parts of the tree to navigate and share their thoughts out loud. This helps researchers understand how easy the website is to use and where participants face difficulties.

The goal is to find any issues with the website's structure, like confusing labels, unnecessary paths, or hard-to-find content. Researchers collect data on how well participants complete tasks, how long it takes them, and what they think of the website.

Once enough participants have completed the website tree test, the collected data is analyzed. This helps identify common usability problems and areas of the website that need improvement.

Designers and developers can then make changes to the website's structure, labels, and navigation paths based on the findings. This improves the overall user experience.

A research report that presents prominent themes along with customer 'pains and gains' and their ideal solutions.

Focus groups gather insights from diverse participants, leading to a deeper understanding of attitudes and behaviours. Through group discussions, participants express preferences and opinions, helping researchers uncover ideal solutions. Follow-up questions investigate reasons behind customer choices, providing valuable insights for businesses to refine strategies effectively.

Focus groups are a research method where diverse participants gather in a group setting to share their thoughts and experiences. This method helps researchers gain a deeper understanding of attitudes and behaviours. The discussions in focus groups uncover ideal solutions and refine strategies. These discussions take place in a controlled environment and are recorded for later analysis.

Focus groups were first developed in the 1940s by social scientist Robert K. Merton. They have since become a popular research method across various fields, including market research, psychology, and sociology.Focus groups are now used in various fields, including market research, psychology, and sociology.

In a focus group, participants discuss a topic or product guided by a moderator who asks open-ended questions. This approach helps researchers gain insights into customer choices and preferences. Focus groups can target specific audiences based on research goals. The findings are summarised in a report, highlighting main themes, participants' feedback, and improvement suggestions.

A comprehensive report and video snippets highlighting the 'frictions and enablers' on the website, along with recommendations for improvement.

Understanding user behaviour, preferences, and pain points is challenging through internal testing alone. Observing real users in a controlled environment helps pinpoint issues and enhance the user experience, boosting website effectiveness.

Moderated usability testing is a vital process for businesses looking to improve their digital presence.

Real users navigate a website or app while a moderator observes and collects data through Lookback – a specialist app uses to record and analyse user behaviour.

This method uncovers insights into user behavior, preferences, and pain points, helping businesses align their digital offerings with customer needs.

Understanding user interactions enables businesses to identify usability issues, prioritise improvements, and enhance the overall user experience. This promotes customer satisfaction, boosts user retention, and can improve conversion rates.

A detailed quantitative test report that offers insights into how users perceive service levels.

Neglecting customers can lead to missed opportunities and errors which cost organisations.

Accelerate your business growth by listening to your customers, the most reliable source of insights about your products and services.

Although accepting negative feedback can be challenging, it's more cost-effective to retain existing customers than acquire new ones. Invesp reports that 90% of customers read online reviews before visiting a business website, and 88% trust these reviews nearly as much as personal recommendations.

Design teams often overlook imperfections that become apparent through real-world use. But research indicates that customers who feel heard develop positive sentiments toward your brand, leading to increased purchases and referrals.

A quantitative test report that evaluates how users rank loyalty and identifies the contributing factors.

Traditional NPS asks only one question and doesn’t explore any deeper insights.

Net Promoter Score (NPS) is a customer loyalty measure developed by Fred Reichheld in 2001. It assesses willingness to recommend a company/product/service on a scale from 0 to 10: 9-10 are 'promoters,' 7-8 are 'passives,' and 0-6 are 'detractors.'

Critics doubt NPS as a sole predictor of company growth, lacking evidence. That's why Insightful offers NPS+, a comprehensive survey that uncovers loyalty drivers.

By identifying these drivers, you'll reveal critical gaps in your customer experience, allowing improvements in services, products, and pricing.

A comprehensive report that outlines the buying personas, drawing from available data.

If you don’t know who your customers are and what they like, how can you expect your services to resonate with them?

Take the time to understand the demographics and psychology of your market segments. This enables tailored messaging to meet diverse audience needs.

Audience Fact Finding will help group your audience, enhancing communication and demonstrating a personalised focus – giving you a competitive advantage.

While some businesses opt for a one-size-fits-all approach to save money, this is a mistake. Serving everyone generically disappoints most and wastes budget.

Segmenting your audience based on shared characteristics can align your marketing, products, and services with your target customers' values and motivations.

A qualitative data report that highlights key themes in how customers perceive value.

Without a clear understanding of what your customers value in your product, you're navigating in the dark, missing out on valuable opportunities.

Customers expect value before committing their loyalty and money. Increasing perceived value can boost customer loyalty. You can gain customer insights and achieve success by offering superior products and services.

The Customer Value Assessment (CVA) involves three steps:

1. Audience: Analyse the market, understand potential customers, identify your target segment, and their preferences.

2. Costs and Benefits: Identify the benefits your offerings provide and their production costs.

3.Competitors: Assess the competitive landscape, pinpoint strategies to differentiate.