Your First Mobile A/B Test Is Over, Now What?

Mobile A/B Testing Post Checklist

Death to the Stock Photo

You just ran an A/B test. What are you going to do next? You could go to Disneyland, but maybe you should find out if you actually “won” with your tests first.

To make sure your mobile A/B testing has been worth the effort, be sure to check these boxes in the aftermath of your A/B test high.

Get Those Automated Insights

We’re sure you’ve heard it before: it’s all about the data. Metrics and analytics are the foundations of your A/B test. Without them, there’s really no point.

We can hear the collective sigh already. We know, who has the time, or the energy, or the willpower to go through reams of data to find what may or may not be relevant for you and your goals.

Enter what we call Significant Changes. If you’ve got Leanplum (or wish you did), in our Analytics suite, we have a section called Significant Changes. The purpose of this section is to demonstrate any metrics that have undergone statistically significant changes.

You don’t have to wade waist-deep through numbers to find what metrics your A/B test impacts. We know that can be time consuming, so our system does the work for you. With Leanplum, you get analytics that highlight the action you need to change.

Mobile Analytics Significant Changes


With Significant Changes, you can quickly see if any metrics are impacted drastically, either positively or negatively.

Set Your #Goals

You have the option to set goals for your A/B test before you run them, so if you’ve set that up, you’ll definitely want to check those out.

But, we know that you may have either A) forgot to set those up, or B) had no idea what kind of goals you wanted to set and left them blank instead.

Mobile Analytics Goals


We get that.

What’s great about the Significant Changes section is that you can use it to determine what you want your future goals to be.

An example: Say you created an A/B test around a messaging campaign. You sent two different versions of a push notification about a new feature deployment and set a goal to increase user engagement by 25 percent using the new feature.

After about a week, you notice a clear winner between the two pushes. Awesome, now you know what type of content will do better in the future. You check out Significant Changes and notice that your average revenue per user changed significantly in the variant (B) version of your test. Guess what: it’s not too late to set a goal for that metric, too.

We know it’s hard to predict all the things that your A/B test might affect. That’s why after you run any test, you can retroactively set goals based on those metrics in Significant Changes. As soon as you set up your new goal, it will automatically populate with historical data, so you can start measuring results right away.

How do we do this?

Leanplum collects all these metrics out of the box, with zero effort from you. As we saw in this example, you never know when you might need to set a new goal based on surprise results.

Mobile A/B Testing Significant Changes


Iterate & Improve

So, you checked the numbers, you achieved your goals, and you set new ones. Now the final step: implementing the knowledge you acquired.

In Leanplum, our A/B testing section has got a handy little “I’m finished” button that is pretty self-explanatory.

a/b testing results

After each test, you can click that button and decide whether or not you want to keep the control (A) or switch to the variant (B) version.

The choice is up to you, but we suggest that you make this decision based on whether you like what you see in the previous two steps (running the numbers and checking your goals).

If you’ve met (or exceeded) your goals and you’ve got a lot of positive Significant Changes (and no major red flag negatives), you might want to go with the variant.

If you didn’t end up seeing the kinds of changes you were hoping for, or you think you could do better with your goals, you can keep the control version and run another test. The choice is all yours.

Mobile A/B Testing Results



Bookmark this checklist and keep it close. A/B tests can be a powerful tool, as long as you know what to do with the data behind them. With mobile automation tools like Leanplum, you don’t have to be a data nerd to be successful. Let our data science crunch the numbers for you.

Leanplum is the most complete mobile marketing platform, designed for intelligent action. Its integrated solution delivers meaningful engagement across messaging and the in-app experience. We work with top brands such as Expedia, Tesco, and Lyft. Ask for a personal demo here.