Mobile apps are always a work in progress. The most effective mobile teams are the ones that have a deep understanding of their customers on a qualitative level as well as the quantifiable data to creatively iterate their way to success.
Leanplum customer Wanelo exemplifies human-centered optimization. The agile team combines data from Leanplum with a deep understanding of what makes happy users to create innovative features that optimize the mobile user experience. No black box algorithm has this level of deep context to outperform a team like Wanelo’s.
Wanelo (from Want, Need, Love) is a digital mall. It contains all stores and products in one place, so that people don’t have to go to each store’s website individually.
Launched in 2012 and headquartered in San Francisco, Wanelo has seen tremendous growth in the past year. In August 2013, the company reported 11 million registered users, up from 1 million in November 2012 with completely organic user growth.
Wanelo users sign up and start following their favorite stores and people whose taste they like. This creates a personalized feed of products just for them. The team started using Leanplum in November 2013 to interpret and act on the results of their experiments.
“Having the right tools to be able to A/B test on mobile allows us to focus on core feature development and push the product forward fast in an agile way,” said Sean Flannagan, Wanelo’s VP of Product. “A tight feedback loop is essential to building something really powerful.”
The Optimization Goal
With Wanelo’s loyal and avid user base, the team is focused on making sure new features increase critical actions like ”product saves per user” and lead to long-term retention.
The center of the mobile experience on Wanelo is “My Feed,” which is a feed of products from the people and stores users follow. The team believed there was an opportunity for My Feed to perform better for key metrics, such as saves. They questioned the content of My Feed and wanted to experiment with some new approaches to presenting users with relevant products.
Wanelo used Leanplum to quickly test three alternate versions of My Feed in their iOS app against the control:
- A new visual format of My Feed with the same content as the control
- A feed of only recommended products based on user’s activity
- A mixed feed of recommended products based on user’s activity + content from My Feed
Each version received 25% of the total traffic with a control running the remaining 25%. The test ran until the optimization goals, saves per user and average session length, received statistically significant results.
The version with all recommended products won decisively with a dramatic increase in saves per user and significant increases in user engagement.
Behind the positive story the numbers told, some users left negative reviews in the App Store because they were no longer able to clearly see content from the people and stores they follow.
User reviews, user-testing sessions and other qualitative feedback suggested that users liked the recommended products Wanelo generated for them based on their saves, but didn’t want these recommendations mixed in with content from the people and stores they follow.
“We saw a massive increase in saves and significant increases in average session length and time in app. Normally this would be a huge win, but you have to look at every available indicator to make sure you don’t get tunnel vision,” said Sean Flannagan.” When we saw negative user feedback, we incorporated that into our thinking.”
As a consequence of the user feedback, the product team acted quickly using Leanplum’s Content Management System to close their experiment and revert to the control version of My Feed.
The Wanelo team knew they had to find a way to surface both recommended and personally saved and curated items without hurting the user experience for either of the two use cases. Having witnessed the value of a feed comprised of recommended products, they set out to find a way to integrate the concept more seamlessly into their app.
The Wanelo team decided to test a new feed separate from My Feed. This feed would contain only recommended products. They called it the Magic feed.
In the interest of speed, the Wanelo team decided to test this new Magic feed in their Android app first. They used Leanplum to enable the Magic feed for 50% of their Android app users.
The version with the Magic feed saw significant increases in saves per session, saves per user and time spent in app. Retention was not negatively affected and there were no other unforeseen negative effects. Some users posted positive tweets and Instagram photos of their Magic feeds: a great indicator that validated the results the Wanelo team was seeing in Leanplum.
As a result, Wanelo saw engagement increase and saves per user jump to one of the highest levels in company history.
With these positive results on Android, the Wanelo team felt comfortable enough to enable the Magic feed on iOS for 100% of users, with the ability to dial it down via Leanplum if necessary.
Right away, ratings and comments from their community were off the charts.
While optimizing for key success metrics should be a priority for any data-driven company, it’s important to avoid tunnel vision and be aware of how new features impact user satisfaction. The product is always a work in progress. Black box optimization will not pick up on all the signals that matter to your team, nor will it empower teams to look beyond the data and creatively come up with ideas for testing.