Three Steps to Increase Your ROI With Mobile A/B Testing

Mobile A/B Testing is only as good as the time you put into setting up your tests, monitoring them, and analyzing the results. Before beginning an A/B test, have a clear goal and purpose of what you hope to achieve.

You could follow the classic A/B test example and change the color of a button to green for half your users and leave it purple for the other half. But before beginning any test, the most important question to ask is how is this A/B test going to increase your ROI or conversions? Besides this, how are you going to measure the impact it has on user engagement or retention?

To help make sure that your testing has the desired effect (and to easily find out why not if it doesn’t), here are some easy tips.

a b testing android

Setting Up for Success

Before running a test, there are three big questions you should answer:

What Do You Want to Test?

Within Leanplum, you can test messages, variables, or the visual interface. You can test each of these individually, or in combination.

For example, if you want to test a new feature in your app, you can choose to swap out an old variable with a new one. But if you’d also like to alert your users to this change, you can bundle a push notification into your new feature A/B test. You could test two different versions of copy, or send it at two different times to see which results in higher open rates. You can test both A/B and MVT.

This can be beneficial because you don’t need multiple services to handle your content (variable changes) and marketing (messaging) needs — you can handle both within Leanplum.

Who Is the Audience?

The default setting for an A/B test is a 50-50 split between your users, but you can change this to any ratio that you’d like. You can also choose to show your variant version to a specific subset of users using segmentation, which you can learn more about by watching our in-depth video on how to filter and group your users.

What Are Your Goals?

Your goals enable you to indicate what your most important metrics are for an individual test. Set these ahead of time if you know the ideal results you’d like a test to achieve.

Whether or not you set goals, Leanplum will still collect and analyze all your metrics. Thankfully, you’re not limited to data for your predetermined goals, and we’ll always show you the metrics that have undergone significant changes on the dashboard.

Mobile A/B Test Like A Pro

We like to provide pro tips for our readers, and this time is no exception. After you’ve handled the fundamentals of your A/B test, try experimenting with these features as well.

Time Estimator

a b testing androidThe Time Estimator shows how long it will take for your goal to reach statistical significance. This way, if you see that goal may take a long time to reach significance (or at least longer than you anticipated), you can reconfigure your goals to see significance more quickly.

Impression Criteria

apps a/b test examplesThe Impression Criteria guarantees that your data isn’t diluted by users who haven’t actually experienced the changes you made in your app. You can configure your test so that a user must trigger a specific event, such as a checkout page, for their behavior to be tracked as part of your A/B test. Behavioral changes are attributed only to those users who have been exposed to the change, and not everyone who opened the app.


android ab testing

Setting targets allows you to specify who experiences your A/B test. For example, you can choose to target only first-time users for a new onboarding flow.

Additionally, you can use the magnet button to toggle “stickiness” on or off. If you turn this on, it means that a user will remain in the test after being assigned, even if they have fallen outside of the specified target group. In the first-time user example, this means that when a first-time user opens the app, they will be added to the initial A/B test. They will remain a part of that test when they open the app for any successive sessions. If this button is switched off, they will only be included in the test for their very first session, and then removed from the targeting criteria for future sessions.

Analyzing the Results

Leanplum’s Automated Insights are backed by statistical significance. In the graph area, the shaded region indicates the low and high bounds of the confidence interval. If the shaded regions don’t overlap, your test is statistically significant.

mobile a/b testing platform

The metric tiles are broken into three main groups. The top section shows the metrics you assigned as goals. The middle section contains your favorites and remains static on every analytics page (regardless of the test). You can configure your favorites in the Analytics center.

use mobile a/b testing to increase ROI

The last section is for Significant Changes. Leanplum automatically shows all the metrics that have experienced statistically significant changes. This is to help you avoid unforeseen consequences and to gain access to more than just the data from your predetermined goals. We like to say that A/B testing is about tradeoffs; tests will always produce a mix of positive and negative results. Significant Changes ensures you’re aware of both the positive and the negative results, so that you can adjust your campaigns and improve your next iteration.

Time to Start Testing

For a more in-depth overview of our A/B Testing platform, you can check out this video for even more information.

Leanplum is the most complete mobile marketing platform, designed for intelligent action. Our integrated solution delivers meaningful engagement across messaging and the in-app experience. We work with top brands such as Expedia, Tesco, and Lyft.

Schedule your personalized demo here.