Activation, retention and more: growth lessons from the industry's best

This week we released The Growth Handbook, a collection of tested frameworks and invaluable lessons to help steer your company’s trajectory up and to the right. In many ways, the genesis of this new book was our podcast series and the enlightening, in-depth conversations we hosted with industry leaders who’ve grown startups to many billions in revenue.

To mark the release of the book, this week’s podcast is a special compilation of some of our favorite insights from those growth-focused conversations. You’ll learn from:

• Andrew Chen, General Partner at Andreessen Horowitz
• Casey Winters, former growth leader at Pinterest and GrubHub
• Shaun Clowes, VP of Product at Metromile
• Gina Gotthilf, former VP of Growth and Marketing at Duolingo
• Sean Ellis, CEO at GrowthHackers
• Brian Rothenberg, VP and GM at Eventbrite

To hear each of these conversations in full, check out episodes of our podcast linked in each section below. You can also subscribe to the show on iTunes, follow us on Spotify or grab the RSS feed in your player of choice. What follows is a lightly edited transcript of the episode.

Andrew Chen on fighting acquisition channel fatigue

Andrew Chen: The idea is that, especially in pure consumer products, there was a period of time where we had address book importers: you got an invite to a product from a friend, and you were like, “Oh my god, what is this? This is so cool. I want to use this.” And people just got used to that. Eventually, we got to a point, especially now that we’ve gone to mobile, where we don’t have contact importers that work as effectively as the ones before. This is also because email spam and text spam are very different things. There are lots of laws around the latter with the Telephone Consumer Protection Act, and intermediaries like Twilio have a very strict stance on that stuff. What this means is that virality is much harder, and the spammy kind of virality we saw during the Facebook days is not there any more.

So, you have a few options: you could work in a different area where these channels haven’t been exhausted yet. My calendar has all the information about whom I’m meeting on a day-to-day basis. The documents I’m editing and everyone else’s edits on those documents tell me who’s interested in the topics I’m interested in. My email inbox is completely obvious. Even some of the other tools like Slack and Asana give great signals on whom I’m collaborating with. But I’ve actually seen very few products that are built on that idea. It’s this workplace graph that’s just sitting there. So, I’m really excited to see how people take consumer ideas, bring them into the workplace and then adjust them. For instance, in a workplace you don’t need to ‘follow’ your coworkers; you’re on teams automatically, you know you’re on the same email domains, and it’s much easier in many ways.

Inevitably whatever worked in the past will no longer work.

The other way, within consumer products, is you have to figure out how to make a lot more money and then use different forms of paid acquisition. If you are a product that figures out an awesome consumer subscription business – or you’ve figured out a high-ticket item like housing or cars – all of a sudden you can innovate within paid acquisition. You can do paid referrals or paid ads. You can figure out different kinds of incentives. On a total side tangent, we’re very early on a lot of the crypto applications, but if we fast-forward a couple of years, people are going to play around with a lot of really innovative approaches, whether they’re referrals or a different kind of incentivized engagement.

Adam Risman: Looking at this from a higher level, eventually there will always be diminishing returns on these channels. That’s an idea developed in one of your most famous essays, “The Law of Shitty Clickthroughs.” In the time since you wrote that, how have you seen that observation materialize in new channels that have emerged?

Andrew: To summarize the idea, the very first banner ad was for HotWired, and it had a clickthrough rate of more than 70%. Now 20 years later, you look at the average clickthrough rate and it’s like .05%. It’s very low, and anyone who has worked in the industry long enough has seen this happen with email, SMS and all sorts of things for a bunch of reasons. You have competition, and you have the platforms themselves saying, “Hey, we need to clamp down on this.” There’s literally habituation from end users who are thinking, “Oh, it used to be fun to get a invite from my friend, but now I’m getting it all the time.” It’s just less effective, because you have a crowding effect.

The reason why I call it “The Law of Shitty Clickthroughs” is that it’s something that has been with us for a really long time and will continue to be. For all of us in marketing and growth, that means we have to continually find the fresh powder, because inevitably whatever worked in the past will no longer work. By the time a case study has been published on Medium about something that works, it’s probably done. Everyone still has to do it, but then you have to move beyond that.

Shaun Clowes on solving your product’s activation problem

Shaun Clowes: It never ceases to amaze me how much time we as a industry spend optimizing our acquisition tactics to acquire a huge bunch of people. When you think about all the energy that has gone into this, and understanding who those people are and how to go and find them, and then you look at the number of people that drop off in the first five to ten minutes, the first day, it always breaks my heart. Even at Atlassian, where I felt like we were getting better and better at this, we had the charts that would show us what was happening. Every one of those people is a person whom you have fundamentally burnt. You have failed to give them what you told them you would give them.

When I think about the ROI of things that you can do in a business, make certain that your customer is safely handed from acquisition to the activation. Make certain that they are activated and you have done everything in your power in order to make certain they have found their “Aha” moment and they have began habit forming. Then make certain that they’re getting the maximum value from your software though engagement. Those are generally very low investment, but with potentially massive rewards. Not only because you keep those users, but if you truly succeed at that, and you get high engagement, then what you really get is sustainable businesses, because you get word of mouth.

If you have $100 and you’re starting up, I’d bet $80 of it in the activation phase.

People who are highly engaged with your software are always the people who love it, and those people who love it will tell other people. The most authentic form of acquisition, by far, all day, every day, is word of mouth. It’s amazing the business you can build once you have that engine going.

Your problem is almost always going to be in activation first. If you have $100 and you’re starting up, I’d bet $80 of it in the activation phase, because what happens is people fail to have their “aha” moment. And even if they have their “aha” moment, they can drop out before it becomes a habit. There’s a huge amount of value there.

The tactics I use in that space are first understanding what my drop off rates are, and then watching users in their very first experience. I try to understand, “What are the buttons they don’t find? What are the things they are confused about?” Then try really simple stuff like a dialog box saying, “These are things you are probably looking for.” Lead them in the most likely three places they want to go. It’s amazing.

At one point we had a 12 step onboarding flow at Atlassian. It was deeply involved, it had a whole bunch of things that it taught you, and it was very successful. But when I talk about champion/challenger, we were constantly trying to beat that, and we later on ended up with an onboarding flow called “Choose your own adventure.” It was one dialog box with three buttons, and it outperformed the 12 step program. The 12 program was trying to tell you enough that you could do the rest, but it turns out that most people arriving at the software wanted to do one of three things. Rather than needing to educate them about those three things, about the way in which they could go about thinking about the software and find it, just giving it to them was enough.

Casey Winters on convincing your users to stick around

Casey Winters: You can send emails and notifications, and that will be a layer of additional retention on top of your product, but it won’t fix broken retention. It’s an optimization play. I can give a few examples of what we found at Grubhub and Pinterest.

At Grubhub we asked users, “Why don’t you use Grubhub more often?” They would say, “It’s expensive.” We thought that was weird, because Grubhub actually doesn’t charge you at all. But, what they meant is that minimums and delivery fees were high, and it felt like they were spending a lot to get dinner. We went back to our restaurants, and we said, “People aren’t ordering as much as they would, because your minimum is too high or your delivery fee is too high. Why don’t you try a test where you drop those, and we can see if the increased volume you’ll receive will make up for the lower margins on those orders.”

At Pinterest the problem was that the product had gotten too complicated.

We got a few restaurants to try that, and they saw a dramatic increase in volume that way more than made up for the loss in margin per order. We were able to use those testimonials to help other restaurants lower their minimums and fees. That increased how many people would continue to use Grubhub, how often they would use it, and how many people would order the first time, because they were able to find something that was affordable for them that they liked. It was a really big effort and win for the company.

At Pinterest the problem was that the product had gotten too complicated. As we did this research internationally, we saw that we were throwing too many concepts at people, and they weren’t seeing content that they liked and understanding the value. So we went back to the product, and we stripped it down. We basically removed all the advanced features for new users and said, “The only thing that they’re going to do is connect to cool content. And then, if we understand that they get that, they like the content, and they start saving it, then we can start to re-introduce some of these other things.”

It’s silly to try to tell a person about a group board if they don’t know what a board is yet. We did that for basically the first 30 days of the product – we were going to make sure you understand that you can save things. We’re going to make sure you can understand that this pin came from another website, and you can go click on it to get there. Everything else is gone until we make sure you understand that. Then we can start to slowly re-introduce other things. That really helped out activation rate and long-term retention for the business.

And we weren’t too cute about either. For example, at Pinterest, the save button said, “Pin it.” We’d ask people internationally, “What do you think that button does?” They were saying, “I didn’t know it was a button. It looked like the logo.” They wouldn’t click on it. They would never find out what it was. We finally said, “Why don’t we just make it the local word for save?” And then, people started clicking on it more, and people started connecting to the value more. We were trying to get a little bit too cute, a little bit too brand-focused, on some of these things. The most important thing a brand should do is help people understand the value of this thing. What does the brand promise? We were obscuring that with our branding work. That was another optimization we made that really helped out.

Gina Gotthilf on testing and experimentation at Duolingo

Gina Gotthilf: This came from FourSquare back in the day, but there’s a lot of different apps and games where you can collect badges as you go. We really wanted to introduce that into Duolingo, but actually when we first tried it failed massively. We thought that badges didn’t work for a whole year, and then decided to try it again and it was so successful. Talking about all of these different drawbacks to other metrics that I was mentioning before, this was one case where it helped retention, it helped monetization, it helped learning, it helped everything, because we could just tell people, “Do this for a badge.”

If you offer badges to people to display different behaviors that are beneficial to them and are also beneficial to the app. For example, use Duolingo every day is an obvious one. It’s not a badge, but if you have an X-day streak, then you earn a badge, so you really want to get there. Use Duolingo in the morning, use Duolingo at night, or click on this tab and engage with someone. Invite a friend. Buy an item. There’s all this different stuff, some of which is related to learning and some of it which is just related to getting the user to see more of the app and experience more of it, which leads them to like it more and retain longer. Also it works for very short-term things too, like invite their friends and buy something on the app.

Adam Risman: You test badges, you sit on it for a year, and then all of a sudden it’s this game changing thing that you added to the app. How did the team do such a 180 on that project? Was it how the experiment was run? Timing? What happened there?

Gina: It was the experiment, and I can credit my team with really insisting for badges way after I said, “You know guys, we shouldn’t waste our time. It’s too much of an investment. We don’t really know what the return is going to be.” When we first thought about introducing badges, we thought about why people liked them. “Maybe it’s because they really like this feeling getting something and feeling rewarded and like they did something good,” we thought, and we decided to replicate that.

We would often come up with a minimum viable test for things. So instead of creating an entire badge system, what’s the simplest thing we can do to test whether that’s going to have an effect or not. If it has an effect then we can go and build an entire badge system, and that’s going to take us months.

When you test something and the results are really good or really bad, don’t rest on your laurels.

We just did this thing where if you signed up you got this pop-up of a girl with balloons, and it was basically a congratulations for signing up thing. In retrospect it sounds really stupid, and I can’t even believe that we thought it was going to work, but we were really convinced that that would replicate that feeling of getting something, and if we saw an increase in metrics there, then we could start including more of those throughout the app and they would be badges.

Unsurprisingly, at least to me now, that did nothing, and unfortunately our conclusion was that it’s not worth investing in badges. Then we spent all this time just trying to shoot for lower-hanging fruit, things that you can test that take less time to develop, less time to design and just require less. Things that might bring us smaller gains, rather than trying to go for these big bets, for a really long time.

The whole time my team kept saying, “Let’s do badges, let’s do badges, let’s do badges, let’s bring it back.” Finally I was said, “Fine, if we’re going to do it this time then we should really think about why that didn’t work, and let’s really invest.” It took two or three months to fully design and implement badges. It seems simple, but where do they live on the app? Where do people see them? When do they get triggered? When do you receive them? How many of them are there? Are there going to be tiers? Can your friends see them? There’s so many layers to the badges, so it took us a really long time. But for us as a team it was probably our most successful experiment. Because we were so excited about it, we spent so much time on it, and it really, really worked out not only for us, but other teams’ metrics too.

Adam: For something like the badges or the streak or anything else you worked on, how important is it to optimize those things over time rather than resting on your laurels? Do you start to see diminishing returns, or when it’s something that big, can you put it out there and move on to something else for a while?

Gina: When you test something and the results are really good or really bad, you don’t rest on your laurels. You don’t just say, “That was really bad, let’s ignore it,” or “That was really good, let’s leave it.” Now that you know that has a really big impact, it’s worth your time going and trying to squeeze as much juice as you possibly can out of it, or improve that feature as much as you can.

With badges, we went on to make tiered badges. Now you can get level one, level two, level three, and one was gold and one was silver. We introduced other types of badges with other types of behaviors. Other teams were asking, “Hey, can you do a badge for getting people to do this,” because they wanted their metric to be helped by badges.

Same with streaks. We spent a long time thinking about what can we do with streaks and how can we make this experience even better, because we know it matters to people. If you try something and you get a whatever, a really tiny percent change, then great. If it’s statistically significant and it’s worth the engineering costs and the code decks that’s going into it, then you launch it, but you don’t go back and keep trying and trying because you already know that it’s not super impactful.

Sean Ellis on finding your North Star metric

Sean Ellis: In order to drive sustainable growth, you need to have a product that people get value from and keep using over time. If you’re focused on expanding the value you’re delivering, you’re going to be able to retain users. Not only that, but what I’ve found in every company that I’ve worked with is that the most important growth lever is pure, natural word of mouth.

By focusing on delivering a valuable experience and measuring value across the customer base over time, you’re looking at the thing that’s going to drive sustainable growth. The North Star Metric is really trying to quantify that value over time. If everything you’re doing is about trying to expand that, you’ll be in good shape for driving long-term growth in the business.

In Sean’s recently revised Growth Pyramid, finding a North Start Metric directly follows product/market fit.

Adam Risman: What are some tangible examples – good and bad – of North Star Metrics you’ve seen?

Sean: Airbnb’s metric is nights booked. If you were trying to optimize everything around app installs for Airbnb, you could potentially be up and to the right and look like you’re killing it. But ultimately if those people are not booking a room on Airbnb, you’re not making any money – and they’re not getting any value to the point that they’re going to tell their friends. So, it’s “nights booked” that actually creates value not only for the guest, but also for the host.

For something like Facebook, an easier proxy for value is just “daily active users.” That’s their North Star Metric. It’s having more users on the platform that ultimately creates more contact for people to interact and engage with other people’s content so it creates that back and forth loop where people are getting value from that system.

Adam: How does that trickle down to the rest of the organization?

Sean: Now each person and team within the company can start to examine their role and think about how to expand that value. The marketing team might be bringing new people in the front door. The product team might be helping activate and retain people. A support team might be helping retain people. People actually start to understand their role in expanding the North Star Metric, which means you get a cross-functional, shared mission. People begin pulling in the same direction, which is hugely beneficial to creating a company that grows over time.

Brian Rothenberg on importance of mission-driven growth

Brian Rothenberg: A company’s mission and growth, and the strategy involved in both, are inextricably tied. People will build the right things if they understand what they’re building towards and how they are uniquely positioned to do it.

I’ll share an example. Eventbrite was all about open accessibility – reach the masses, empower the masses, and democratize ticketing. These are terms that our founders would frequently talk about. People understood that this is core to our ethos.

A company’s mission and growth, and the strategy involved in both, are inextricably tied.

The product was initially launched to be only for paid events, but some of the early users were actually putting zero dollars into the price field for tickets. Some companies would have viewed that as a loophole. They’d have closed that loophole and said, “We’re just going to monetize.” Our founders said, “No, this is happening. It’s organic, it’s open, and it matches our mission,” and they let it happen. That ended up building one of the biggest growth levers for the company, which is freemium.

A huge percentage of our tickets sold are actually free tickets, but that drives trial. A lot of those initially free event creators convert over to paid. It’s a huge lever for us. That never would’ve come about if it wasn’t tied to the company’s mission.

I like to view a business as an onion. You’re constantly peeling back the layers to more deeply understand it. Even if you deeply understand your business, you have to always keep trying. Then, if you feel like you have your business down, you have to be peeling back the layers of your customers’ motivations. They’re so tied together.