The Best Way to Launch a Push Notification Test

Running a push notification test is hard when you’re not aware of every variable. It’s easy to say that you want to A/B test your content, but can you sit down and list out every element of a push notification that you ought to test?

Let’s say you’re drafting a series of push notifications as part of a new lifecycle campaign. You’ve read about the importance of A/B testing, and you want to optimize your messaging by delivering the content that resonates best with your users.

This is a good idea in theory, but what does it entail? What should you test to optimize your push notifications? Is it better to simply test everything?

To answer those questions, we created a step-by-step guide that lists out every major push notification component you can A/B test along with an example of when you should test it. This is the best way to launch an end-to-end push notification test. If you’re ready, let’s dive in.

Message Copy

push notification test

What Is It?

The most obvious element of a push notification to A/B test is also one of the most important. The copy is the message — it’s the reason why you’re reaching out to your users in the first place. Without well-written, relevant copy, even the most timely message could be regarded as spam.

Consider experimenting with tone and message length in your variants. You can tweak your tone of voice with anything from word choice to emoji usage. It’s also difficult to predict what message length will work best for your users until you test it.

When Should I Test It?

If you’re developing a new push notification strategy, it could be a good idea to A/B test your message copy before introducing dynamic factors like personalization. This will help you develop best practices that are specific to your audience to guide you through future campaigns.

At the same time, if your personalized messages aren’t providing the results you’d like, it may be a good opportunity to go back to the drawing board and A/B test your copy.

Rich Push Notifications

push notifications what to test

What Are They?

Rich push notifications were first introduced in iOS 10 and Android 4.1. Compared to traditional text-only push notifications, rich push messages allow mobile teams to include images, audio, and other media. It’s also possible to customize the message buttons and layout, as shown above.

Rich push presents a big opportunity for companies looking to increase push notification engagement, but it’s not a one-size-fits-all solution. Users don’t always want to load an image or a gif with their push notifications, especially if text would have worked just as well. Deciding when and where to use rich push notifications is a good reason to run a mobile A/B test.

When Should I Test Them?

If your push notification engagement is in a slump, try experimenting with rich push notifications. Adding some variety to your messages could give them an engagement bump.

Even if you already use rich push notifications, you can compare different types of media, like images versus gifs. Rich push virtually guarantees that you’ll grab the user’s attention, so you can iterate on it to find the media that best converts user attention into taps.

Audience Segments

how to test a push notification

What Are They?

Putting aside the push notification content, there’s another factor that’s important to optimize: who you target. Rather than sending messages to all of your app users, you can target your push notifications to audience segments. These segments are groups of users filtered by a given criteria, such as app usage time or in-app events triggered.

It is tempting to test your push notification content on your whole audience and make generalizations from that, but this approach may not tell the whole truth. For instance, what if rich push notifications perform well as a whole, but a specific segment of your audience never reads them? You run the risk of losing these users if you keep sending them rich push notifications, even if that strategy works well on average.

By A/B testing your push notifications on different segments, you can customize the content that each user group experiences.

When Should I Test Them?

Once you’ve established a few best practices for your overall message content, consider delving into your app usage stats to determine if any segments of your audience respond in a surprising way.

The reason for doing this after settling on your overall message tone and content is because it’s difficult to study audience segments without data. Once you have a body of data that shows that your audience is, on average, responding to what you’re saying, you can look into the details to find whether there’s an underserved segment that’s responding poorly.

Message Timing

push notification testing

What Is It?

Timing is another non-content factor that’s critical to test. It might not sound important, but the data shows that 63 percent of push notifications are sent at the wrong time, while well-timed messages can enjoy 800 percent more opens.

There are two ways to personalize message timing: behavioral triggers and algorithmic scheduling. The former means that users are sent push notifications when they meet given criteria, like not launching the app for seven days or adding an item to their cart without checking out. As cited above, messages sent with behavioral triggers experience an 800 percent higher open rate, so there’s little doubt that triggered messages are effective.

However, not all messages can be triggered. What if you want to announce a new app feature or a promotion, regardless of the user’s actions? That’s where algorithmic scheduling comes in. Leanplum’s solution to this problem is Optimal Time, a feature that automatically sends push notifications to individual users when they’re most likely to open.

Since Optimal Time is a machine learning algorithm, it goes one step further than simply taking time zones into account — it adjusts its timing on the fly, based on each person’s app usage history.

When Should I Test It?

Timing is situational, so a lot depends on the type of campaign you’re running. For behavioral push notifications, you can try A/B testing slight variants of each trigger. For example, if you’re sending a re-engagement message to activate dormant users, you can start by sending the notification after a week, and then test it against a campaign that sends the message after 10 days.

For non-triggered messages, it may be best to start using algorithmic scheduling right away. Of course, it’s never a good idea to trust a black box, so we always encourage Leanplum customers to A/B test our Optimal Time algorithm. If Optimal Time returns better results, there’s no harm in using it right off the bat.

Message Channel

run a push notification test

What Is It?

A messaging channel is a means through which you can communicate with your audience. For mobile teams, the most common messaging channels are push notifications, in-app messages, emails, and persistent messages stored in the App Inbox.

The same message delivered through another channel could yield drastically different results. Consider a discount code that you offer to your most engaged users: since the recipient might not use the code right away, it’s possible that the code will get lost if it’s shared as a push notification or in-app message. That same offer could yield a higher conversion rate if delivered via email or App Inbox, enabling users to refer to it when needed.

When Should I Test It?

The messaging channel fundamentally affects how the content is received, so iterating on it is not as simple as testing minor edits to the copy. The best time to test an alternate channel is when you feel that a broad, strategic change should be made.

In many cases, it’s equally acceptable to send a message through any of the available channels, but one particular channel might resonate with your following. If your message engagement is lagging across all user segments and you can’t identify why, it might be that you’re using a suboptimal channel.

By testing the same message on different channels, you can decide if one of them should be your “default” or primary channel, or if your messages should be shared equally across all available channels.

These are the five essential components of a push notification test, but there are several smaller factors within them that can be optimized. What attributes should I use when making an audience segment? Do some events work better as behavioral triggers than others? And what are the differences in rich push notification engagement between iOS and Android?

The good news is that these questions can be answered through the same techniques described in this post. With this in mind, you’re ready to launch an end-to-end push notification test.

Leanplum is building the marketing cloud for the mobile era. Our integrated solution delivers meaningful engagement across messaging and the in-app experience. We work with top brands such as Expedia, Tesco, and Lyft. Schedule your personalized demo here.