Andy Carvell joined SoundCloud in 2012, when the company was just over 80 employees and 10 million monthly active users.

The service now boasts over 150 million registered users, and monthly actives in the high tens of millions.

I recently spoke with Andy as part of a 1 hour interview covering:

How he brought a web-first product to mobile
Activity notifications, rich push, and other techniques for driving mobile growth and retention
Andy’s “Mobile Growth Stack” for 2017

You can watch the full interview here, and check out the truncated text version below.

During Andy’s four and a half years at SoundCloud, he worked on a range of mobile growth topics, and in a bunch of different team configurations. He led an international team for a while, focused on running experiments in new markets, and did growth work breaking ground in Latin American and other developing markets.

Andy’s last role just before leaving SoundCloud was running the user retention team there. He focused on driving retention of users, both new users and driving more repeat and returning user. SoundCloud has notoriously high user retention, so I wanted to dig in to find the most useful takeaways from the years of experiments that Andy instrumented from the group up.

The biggest retention questions at SoundCloud

Around late 2014 to early 2015, the SoundCloud team identified retention as an area of improvement, especially on the mobile side. It actually wasn't too bad at SoundCloud, by industry benchmarks, Andy told me. But, retention can always be better.

“If you can improve retention a little bit, it's generally better for longer term growth than just increasing acquisition a little bit.

The north star metric for SoundCloud was listening time, and we had some fairly aggressive targets as a company, in terms of increasing listening time overall. You can increase listening time in two main keys: first by bringing people back for more sessions, or second by helping them to achieve longer listening sessions.”

There were lots of different ways the SoundCloud team could move that key metric of listening time, but everything revolved around bringing people back and giving them more reasons to stick around and listen. According to Andy, they were laser-focused on building habits with users so that whenever they were listening, they would be listening on SoundCloud, rather than on the radio, or with a different music streaming service.

A nuanced definition of “retention” at SoundCloud

Retention means different things for different companies, and SoundCloud was looking at a potential additional nuance to retention on the mobile side. At SoundCloud, says Andy, user retention was “listener retention.”

“We aimed at trying to get people not just to come back, but to come back and listen. We didn't count a user as retained if they just came back and relaunched the app again, but actually we wanted them to come back and listen for another session.

The metric we were trying to move was monthly active listeners, indicating month-on-month listener retention.”

A monthly active listener meant that that person came back and listened once within the month period. Andy’s team were also interested in increasing daily active listeners, and trying to increase the frequency of visits during each period. To that end, they designed initiatives that looked at what could bring people back, not just one time in that month, but multiple times in a week, and continue to increase their frequency from there.

The starting point for tackling retention issues

There are many ways to start in on a retention problem. Some teams may want to look at product issues, while others want to jump into messaging and notifications. But Andy and the team at SoundCloud believed that the first challenge was to onboard users well enough that they were able to experience value and understand the product in the first session.

“From there, it was about giving users a good enough reason to come back and listen for another session. But, retention really starts with onboarding; if a user activated and had a good time with Soundcloud in the first one or two sessions, then the barrier for them to de-install the app or just never to come back was very low.”

Benchmarking retention against industry standards

When you’re setting up retention experiments, it can be helpful to know what you want to be aiming for. Andy’s team took a more nuanced approach to industry benchmarks, looking at not just industry averages, but at specific categories.

“It’s particularly relevant for mobile when you consider that in the app store, categories are so different and each category’s retention profiles is different. It's not always very meaningful to look at a blended industry average, but rather to try to look at category average benchmarks for these kind of things.”

Their team mined for data from providers like Adjust, which publishes retention data by category, and covering every category in the app store, and SimilarWeb, which provides data for both web and for Android. this as a service for both web and for Android.

What were the “great,” “good,” and “bad” benchmarks that SoundCloud stacked itself against?

“I'll start with a caveat that I can only talk about user retention here, not listener retention. It's important clarification, because these more generic publications don't go down to that level of specificity -- user versus listener -- which is something that was a really important, but internal, distinction for SoundCloud. So, on a user retention level, looking at day one retention in the music and audio category, anything above 35% is generally considered decent.

Then month one retention, above 45-50% would be considered good. Again, this is across the whole category. At SoundCloud we wanted to significantly over-index versus the category.”

Activity notifications identified as one key to retention -- but with caveats

So, Andy’s team was focused on retention, and the big north star metric was listener time. There are multiple ways to influence that: you can bring people back more often, or you can deepen their engagement once you have them back.

Their team was trying to be in the top 1% of these retention metrics, and identified activity notifications, or push notifications, as something that could majorly influence that.

“I think it's fairly widely acknowledged that push is an effective channel for mobile, if you get it right. A lot of companies don't get it right. We also realized it was not a foregone conclusion, that we could just start sending some push notifications, and we'd get a great result.

A big part of why we chose this project, rather than for example working on the product onboarding, which I think could also have been very impactful, was that at the time, there were no notifications being sent out at all due to the state of the SoundCloud iOS app at that time. It had been re-engineered and re-released a little while previously, and as part of that re-engineering effort, there were no push notifications set up. There were some email notifications, from a very old legacy system, with no measurement around them. We figured they were probably driving some impact, but we hadn't got it instrumented.

We actually even had some users writing in to support, saying, "Hey, where are the push notifications?" We were asking for permission, so we were actually getting a lot of opt-ins, but we weren't sending any notifications. To me, that was a great place to start because it was a blank canvas.”

It seemed like a good place for the team to start, not only because it was low hanging fruit but also because they would be going from zero to X, and the impact could be that much more meaningful.

On Android, Andy explains, the situation was a little more complex because there were notifications happening there already. However, these were not push notifications -- they were client side notifications. They look exactly the same to the user, but they were being generated in the client.

Andy admits that they had very little measurement instrumented around the Android notifications but knew that they could probably drive some additional impact with push. The catch was that they’d have to AB test it against the existing client side notifications, with additional complexity around handling the rollout up to production.

“We decided that what we start with was building the backend notification service, and start with iOS push as the first channel. There we have a really, a clean slate, and somewhere where we can really establish the value of push notifications, and the value of the notifications that we're building. Then we can look at bringing that same system to Android, where we have an encumbered system that we might want to replace, or at least benchmark against.”

Get My Growth Essays, One Email Per Post.

Watch for negative signals when rolling out retention experiments

In addition to the lack of notifications on iOS (but a clean state), and the client side notifications on Android, there were a few other challenges that the team faced.

First, any feature where you’re interrupting the user with a notification is inherently challenging because that communication is potentially annoying and unwanted -- especially if you’re introducing new communications that didn’t exist before.

“If you're sending someone a push notification, I always like to use the analogy, it's like you're coming up to somebody and tapping them on the shoulder. You'd better have something good to say.

If you're going up and constantly tapping someone on the shoulder, and particularly if you don't have anything good to say, they're going to get pretty annoyed. In the best case, they're going to disable push notifications, and in the worst case, they're going to de-install the app and trash-talk you to all of their friends.”

The SoundCloud team were cautious about negative signals and negative impact, and looked for ways to gauge and measure them as they continued to roll out new notifications.

“Some of the negative signals we cared about were opt-out rates, and negative user sentiment. We would ask the community team to really keep an eye out on Twitter and these other kind of social channels, and also in their email support channels, to watch for people complaining about these things and let us know. We wanted to have that negative feedback signal early, ideally before users were opting out. In the early days we didn't have fine, granular notifications preferences where we could see people opting out of specific classes of notifications. That was something we built as a stage two, precisely for this reason.”

Retention is not engagement

There's another interesting lesson here. SoundCloud had decent retention and had gotten that far already without push notifications. The lesson is that retention is a really hard thing to solve for, and push notifications, or any type of notifications, whether it’s email or something else, are not the answer to retention.

Engagement and retention are two terms that are really inextricably linked. You can't have retention if you don't have an engaging product.

“It really begins with product. If the core product is not delivering engagement and value for the user, then nothing else you do around that in terms of sending push notifications or emails or great community support, none of those things is really going to solve that core traction problem that you're going to have, because users won't stick around just because you're sending them emails. They have to get some real value from it first.”

Another way to think about the difference between retention and engagement is this:

Retention is more of a breadth metric: how many people come back within some time period?

Engagement is more of a depth metric: how deep is this experience and how involved is the user with it?

You can't have retention without engagement, and it’s hard to have engagement without a great core product experience, but each one is a distinct area to work on.

The SoundCloud experiment framework

Andy’s team had identified multiple challenges, but were already at millions of users at this point. Because of that scale and complexity, they weren’t going to say, "Okay, we’ve built these ten push notifications. Let's push them live. Let's go."

Instead, and across multiple stages, they built out a system to identify the right push notifications to implement at the right time. What were the phases of that rollout and how did they build a systematic framework to identify, de-risk, and execute new retention experiments?

“When you have an existing user base of substantial size, there's some level of caution you must exercise. You want to make sure that you're rolling stuff out that's going to have a positive impact.

To that end, the first thing we built was an experimental framework around these notifications that we could test not only individual notifications with a test group, and have a control group that was a holdout that never received that notification, but also that we could test combinations of notifications together, to look for possible combinatorial effects that we hadn't considered.

We defined about five notifications to start with, including a “new track” notification, and some social ones like “someone followed you” and “someone liked your playlist.” The other initial notifications we tested were around re-posting, which is another core activity in the app. Basically, we were listening for activities that were happening outside of the user's direct usage of the app, but that they might be interested in and that might bring them back into the app for another listening session, for example.

We defined experimental groups for these notifications, something around 5% of the user base for each experiments, plus 5% for a control group. We had a big group that weren't receiving any notifications during this initial test period, which was around two months long. The first month was all about building the core service, then we did some experimenting. Very MVP.”

Andy and I covered a lot more topics in the full interview:

Kicking off a growth function at SoundCloud 5:00
The retention metrics that mattered at SoundCloud 6:45
Retention benchmarks by category 9:00
The difference between retention and engagement 17:00
Lessons learned in designing retention experimentation, and some surprising factors that affected listener retention 19:30
Breaking down month on month retention by segment 23:00
Other key proxy metrics that rolled up to retention 29:15
Push notification open rates vs attributed opens 30:00
The user psychology behind push notifications 36:32
The mobile tech stack: build or buy? 39:00
The (New) Mobile Growth Stack for 2017 42:10
Andy’s findings on rich push notifications and the impact of emoji on open rates 47:00

Watch the full interview:

Special thanks to Andy Carvell for joining me for this interview. You can follow Andy's extensive writings on mobile at The Mobile Growth Stack on Medium.