Multivariate testing is a technique for testing a hypothesis in which multiple variables are changed. The goal of multivariate testing is to determine which combination of variables performs the best out of all possible combinations. These variables could refer to any type of app content, such as message copy, visual layout, or screen flows. For single-variable testing, see A/B testing.
Multivariate tests must be designed carefully to obtain accurate results. There are countless variables that can be tested against each other in multivariate tests, from in-app UI (buttons, text, etc.) to messaging (audience segments, messaging channels, etc.) and channels (push notifications, email, etc.). For example, if you were marketing a retail app, you might consider testing the placement of an ”add to cart” button to measure conversion rates. You could then use these results to optimize the app further.
Multivariate testing works best with experienced marketers and developers who have mastered the art of A/B testing. The key difference is that several changes are tested against each other at the same time (for example, call-to-action button colors and shopping cart placements). The results help marketers analyze the advantages of groups of variants over others, rather than comparing each variable in isolation.
Multivariate testing should only be used on sufficiently sized audience segments. More in-depth analysis invariably takes longer to complete. Marketers should be aware that the more variables included in a test, the longer it will take to complete. Still, with efficient testing, app users will benefit from improved usability while marketers should start seeing increased conversions and customer retention.
By enabling marketers to study multiple results, multivariate testing can bring about significant changes that may result in higher conversion rates long-term. Retail app Wanelo, for example, ran extensive tests with Leanplum to measure average session lengths and product saves per user. The changes made after these tests resulted in some negative feedback from users, which then inspired a new app feature, the ”Magic Feed”. The new feature saw a 27 percent increase in product sales as well as improved saves per session and average session length.