An ultimate guide to A/B testing on prototypes

Have you ever had an argument with your sibling about which is the shortest route to school?

Well I had it. A fierce one. Really fierce! And the only way to agree to what the other says is to test the two variations.

That’s exactly what A/B testing is — At its simplest, A/B testing is a method for comparing 2 versions of design elements against each other to discover which is the most successful one. So simple? Not really! Rookies often struggle with when should it be done, which tool to use, how to set up the test, and how to know when it’s done. In this article, we will take an example of an app and walk you through the complete process of A/B testing. In the course, we would discover tools, setting up the test and every minor details.

How helpful is A/B testing on apps in pre-LIVE phase?

Honestly, A/B testing is a must at every stage of the project. Then, why is there a special emphasis on A/B testing in the design phase? The reason is, the more you flush out the not-so-great-ideas before the launch of the app, the more are the chances of your app being a hit! Moreover, when the initially launched product will be brilliant, the further iterations would only make it more lovable!

Before we start!

I have my paper sketches ready to be tested on! I have sketched out screens for an eCommerce app.

Paper sketches of an ecommerce app

Setting up an A/B test on your prototype :

I have chosen the paper sketches so as to emphasize on the fact that A/B testing can be done as early as the idea to paper sketch phase of your project.

We will discuss the framework of setting up an A/B test on prototypes.

1. Define Goals :

When we talk of improvements in a project, the scope has no boundaries. So as a project owner, you need to define an area you wish to explore and, ultimately, try to improve. Picking an area does not have to be purely out of intuition.

Conversion funnels on CanvasFlip might help you understand the drop-off ratios and conversion rates. This goes a long way in determining which area needs immediate attention.

Talking of the prototype I have made, I notice that the prototype has maximum drop-offs on the screen where they have to make a decision to BUY a product. The Call to Action I used is “BUY NOW” sharing space with “Add to Wishlist”.

Conversion Funnel of the Prototype

So, based on my intuition backed by data, the goal of my A/B testing is to create a variation for the BUY NOW button or text.

2. Collect Data

Observing the user behavior to collect data is the wisest thing to do! Utilize visitor behavior analysis tools such as CanvasFlip to get user behavior insights on prototypes.

Getting back to the prototype I created, I collected user behavior data around it. I have few session recordings of users using the prototype. Watching these recordings one can easily deduce which area needs variation..

3. Generate hypothesis

Once you’ve identified a goal and collected relevant data around it, you can begin generating A/B testing ideas and hypotheses for why you think they will be better than the current version. Once you have a list of ideas, prioritize them in terms of expected impact and difficulty of implementation.

Brainstorming with your team might be a really good option at this stage. You could pick the top 1 or 2 hypothesis and test them against the current solution.

Lets say, I brainstormed for a hypothesis and my team believes that having a single call to action at the footer would convert more users. Brilliant! So I would create a variation with this hypothesis and test it against the existing solution.

Brainstorming over options for A/B testing

4. Create variation

Creating a variation of the design might be changing the color of a button, altering the copy writing, swapping the order of elements on the page, hiding navigation elements, or something entirely custom. A good practice is to maintain the phase of project you are working on. Let me define what I mean by phase — If you have the existing prototype with paper sketches, design the variation on paper. Same is the case if you have a wireframe or a Photoshop/Sketch design.

So, I created a variation for my prototype as well!

A single call to action vs split call to action

5. Run the experiment

Kick off your experiment by recruiting a set of users. Needless to say, these users should closely match your user persona. Randomly divide your user set into two halves and send out one prototype to each set. Patiently wait for users to use your prototype and participate in the experiment. Their interaction with each experience is measured, counted, and compared to determine how each prototype performs.

6. Analyse the result

The results are in! You’ve done your A/B test, checked that it’s statistically significant, and now you have numbers.

Your A/B testing software (I use CanvasFlip) will present the data from the experiment and show you the difference between how the two versions of your page performed.

Even when the data shows you haven’t made a significant improvement, you’re now in a vantage position than before as you can confidently state what works and what does not.

7. Implement the winner!

If your test has been successful, the decision is up to you. You may want to proceed with the new version or stick to the original one. Or, if you did a small test to begin with, you may want to do another test where even more people are shown the variant.

Final Words

The same test would have been done when the app was live, but concluding the same right at the design phase saves hundreds of dollars that would have been spent on development and post launch usability testing. This is the reason I prefer running plenty of such tests before the app goes into the development phase, and so should you or any other design team do!

P.S. — We always love hearing stories of experimentation and improvement. Do you have a good one to share? Let us know in the comments which elements you test on your website or app.

Related Posts
Vipul Mishra