Metromile's Shaun Clowes on activating and retaining users

The reason is simple: No matter how many new users you acquire, they won’t grow your business unless they’re successfully onboarded, experience the value of your product, and keep coming back again and again.

Shaun is the VP of Product Management at Metromile, a pay-per-mile insurance product that aims to disrupt the way auto insurance is sold; however, he’s best known for his previous work as Head of Growth at Atlassian. Shaun joined the makers of JIRA, Hipchat and Confluence in 2011, where he built the growth team from scratch. By the time he departed in fall 2017, the team had a headcount of 50 and Atlassian had 85,000 paying customers.

Shaun recently joined me at a speakeasy we hosted during the week of Dreamforce to record our very first live podcast. Our chat covers how the role of a growth team has evolved in recent years, why activation is an ongoing challenge, his most successful retention strategies, and much more. If you enjoy the conversation, check out more episodes of our podcast. You can subscribe on iTunes or grab the RSS feed in your player of choice. Below is a lightly edited transcript of the chat.

Adam: Shaun, welcome to the show. To get us started, give us a rundown of your time at Atlassian and what you’re doing today at Metromile.

Shaun: When I joined Atlassian I was running the JIRA agile product. Basically we were building the agile interface into JIRA, the well known software development product. At the same time Atlassian was bringing this consumer model to enterprise software, and we could see that the consumer SaaS businesses were starting to do this thing called “growth.” We thought, “That’s interesting! Can you really do small tweaks that massively increase your growth as business? Can that be applied in our type of business?” We decided we’d give it a go, and I ended up leading a team of five people. We were this little band of merry people trying to change Atlassian from the inside out.

We had analysts, developers and myself, and we started from zero. We had to build it all, figure it all ourselves, and see whether you bring growth to the enterprise in the B2B SaaS space. In four to five years we built out everything you could imagine a growth team needs. We built out all the analytics of Atlassian, Voice of the Customer and an experimentation platform, and shipped thousands of experiments to find where Atlassian was losing customers and getting as many uses as we possibly could.

Adam: Give me a sense for where Atlassian was at a company when this endeavor started and where it was when you left.

Shaun: When I first joined Atlassian, there were about 300 people, and when I left there were more than 2,500 people. That was a massive growth trajectory. When we started growth, we started with five people and by the end we had over 50 people in growth, which probably made us by far the largest B2B growth team in the industry. That was because we built this engine, that could do it all, that had the Voice of the Customer, analytics capability, experimentation capability and the product management necessary to actually start shipping that whole experimentation road map. It really made a difference in the business. We became a strategic pillar, just like a product in Atlassian would be. Just like there was JIRA and Confluence, there was growth.

Establishing a path toward authentic growth

Adam: To set the foundation for this whole chat, what does meaningful growth mean to you and what does the mission of a growth team look like? What were you really tasked with doing?

Shaun: Meaningful growth is a challenging one, because sometimes when people talk about meaningful growth, they mean it in the sense of, “is growth inauthentic?” There is 100% the concept of inauthentic growth and growth teams that are doing it the immoral way that gives growth a bit of a (bad) name.

Growth teams discover why people fail to capture value from your software and connect them back to it.

But, the core of what’s going on there is actually really, really valuable and important, because when you ship products you are fundamentally shipping the solution to problems. And you get product/market fit when it turns out there are people who have that problem who can find your software and solve that problem using your software. Product/market fit is critical. Authentic growth happens from connecting people to your software, but the bit that’s missing from that whole argument is the fact that there are a bunch of people who, for a variety of different reasons, either failed to find your software, failed to successfully use it, or failed to successfully capitalize on all its capabilities.

For me, growth teams are fundamentally about the search process of discovering why it is that people fail to capture value from your software, and connecting all of those people back to it. Sometimes they do big things, sometimes they do small things, but it’s all the same. It’s always this view of, “Why is the customer failing to get the value that we aim to deliver as a business and as a product?”

Adam: Once we acquire these customers, it’s “make sure that they not only discover that value quickly, but frequently too.”

Shaun: 100%, and deepen their engagement too. When you think about software, it’s funny how most product teams have a tendency to think outside in. In other words, What are the problems that exist at the start when they start building a product? They ship a product, and then they start thinking inside out. Now it’s, “What features can I deliver to the marketplace?” But the customer’s still outside in. They still have none of your context, none of your understanding, and they’re trying to leverage this monstrosity of of software that, over time, you add feature and feature and feature to. They fail to capture all this value, and you massively overshoot their ability to capture the value. You don’t even realize you’re doing it unless you have some way of constantly inspecting, understanding, thinking, retrying and iterating on that whole process.

Adam: It’s like you’re acquiring a form of debt, similar to how you would acquire technical debt, but really it’s future debt.

Shaun: That is exactly how I think about it. I understand that shipping features that are not used leads to a form of technical debt, and I get that it’s a painful thing, but it pales in comparison to the cost that you pay when a user looks at your software and does not use a feature. It’s not as though they can’t see it. It’s not as though they’re blind to it. They see it and they start thinking to themselves, “I don’t know what that is. I’m scared. I’m worried. I don’t know if this software is right for me anymore. It looks really complex. It looks really hard. I might just not use it anymore.” Don’t worry about the engineering costs, worry about the fact that you are, literally, making it impossible for your user to capture your value. It’s just a massive problem in all software as you grow past your MVP and into product/market fit.

Defining product/market fit

Adam: You mentioned product/market fit. Most people say you’re ready to grow and invest in growth when you have product/market fit, but, to you, what does that actually look like? How do you determine that you’re there?

Shaun: I like to lean back on something that Sean Ellis once wrote. He talked about three golden questions, but the one that really resonated with me was that you want to ask your customers – a group of people who are presumably retained because they care about your software – a question. “If this software did not exist anymore, would you be sad, and how sad would you be?” You want some portion of people who are in your target market to say they would be gutted, or they would be substantially affected by that. It doesn’t have to be everybody. It just has to be enough people that clearly have captured the value, can see how it changes their life, and that would be sad if it ceased to exist. That’s the core that you can build out from.

When you’re a startup it’s going to be very hard for you to model LTV successfully.

Adam: There are a lot of metrics typically associated with product/market fit. Lifetime value (LTV) for example is something that Casey Winters, a Growth Advisor at Greylock Partners, speaks about a lot. But, that really simple question is a very relatable way to look at it.

Shaun: When you’re a startup it’s going to be very hard for you to model LTV successfully, and to have any idea if it’s real. There are these constellation of things that you can do. The question is, at the point of your development, what are you ready for? What will give you the signal that you need, at that time, in order to give you something to proceed with? Casey’s thinking is always really advanced, and I think he’s a really great writer on this topic. So the question is, “Can I apply those, or do I need to simplify a little bit more in order to deliver something that I can use?”

The role of growth teams grows up

Adam: Metromile is a significantly smaller company than what Atlassian had become by the time you left. They hit 85,000 paying customers earlier this year, so I imagine the types of problems you’re tackling now are quite different. With a smaller customer base, you probably need to test larger things to measure their impact. Are you going back to first principles now? Do you feel like you’re back to where you were at the beginning of your Atlassian days?

Shaun: The advantage we have now is there are a bunch of principles, procedures, technologies and approaches that the industry has now discovered, so it actually doesn’t feel the same. I don’t have the same sample size for various experiments, so experiments are going have to use a bit more directional information.

What is different now is that it’s just so much clearer about how to build a growth program, how to execute one and how to measure one. When you bring that type of toolbox to a product like Metromile, (it’s different). It’s actually quite complex for consumers to understand, because while it is pay-per-mile insurance, the process by which you enroll in this thing, understand what it is, how you’re going to be charged, how you’re going be retained, all of those things are actually incredibly complex problems to communicate and challenges to communicate. Yes, there are some things that I lose compared to Atlassian, but there are a lot of things that I gain in terms of it being a very interesting problem space, and a bunch of tools that I can bring to the space.

Adam: Tell me more about those bigger trends that have emerged in the years since you started the Atlassian growth team, in terms of how software companies are tackling growth. Have we gotten past the point where you’re having to explain that silver bullets don’t exist?

Shaun: Magic bullets are less of a thing that people are pushing for, and in general people just understand the concept of A/B testing and stat significance and the idea that it’s important to do this. You can circumvent some of the debate, but the more valuable stuff is that when we started Atlassian we had to build all of our own everything. We built our own analytics system. We built our own experimentation system. We built our own measurement system. We built literally all of it from the ground up. Today, you look back on that and think, “That’s insane.” You just wouldn’t do that.

There’s a bunch of open source and available SaaS software that you can leverage now, and I think, more importantly than just the technology stack, we now have more advanced thinking on what we need, what a program looks like, how you build one, how you think about growth strategy, and the way in which to think about the problems. That’s hugely valuable, because that allows you to start with nothing, avoid the whole push for silver bullets and say, “At this stage in our development, we need this tactic, and then we’re going to start the ball rolling down the hill, and when we reach month three we’re going to roll this in, and month six we’re going to roll this in, and month 12 we’re going to roll that in.”

Starting with activation

Adam: A lot of growth teams will look at small optimizations first, but where do you think the problems they should really be tackling right away are?

Shaun: The blank page problem, which I think of as fundamental deactivation. It’s the handover between the marketing side that has sold you all this amazing goodness, to your first experience actually in the product. That is this yawning chasm that exists in our industry, and it’s just amazing that it continues to persist. You have worked out exactly what it is the user needs to hear, that’s why you’ve acquired them. You’ve worked out exactly what their problems are, because you told those to them in the marketing acquisition funnel. Then, as soon as they click “try,” you land them on this empty page. It’s insanity.

They have been shown these screenshots, shown these values, told “This is going to be so amazing.” Now you use different language, and you don’t give them any guidance about how to succeed. It’s just like accelerating people into a brick wall and really substantial problem that we face.

Activation is where the vast majority of your initial value is going to be picked up.

For growth teams these days, my advice is that activation is where the vast majority of your initial value is going to be picked up, and the simple way in which you’re going to go about doing that is to carry the promises you made from marketing through into the software. That’s all you have to do. You have to carry it through. That’s not easy. It’s not trivial to work out how to do that in very complex software. But if you have that as a guiding principle, and then you begin your iteration cycle of trying to work out how to do that, you will definitely strike gold. The customers are there to get that value. They’ve decided they believe in you. It’s up to you to meet that brand promise that you’ve made.

Adam: So what types of conversations can a growth team member have with marketing or sales to make sure everyone is on the same page, and when that baton handoff happens, it doesn’t feel like the customer is experiencing that disconnect you mention?

Shaun: That’s why an effective growth team that’s working in the activation space really has to be tightly aligned and coupled with marketing and with sales, because they really need to understand the promises that are being made. From there they can come up with a set of hypotheses to test about how to pass that message through. But, what tends to happen is that those groups actually just don’t communicate to each other.

It’s like this handoff moment that happens at the try button, or at the sign up button, and from the customer’s perspective, you’re shipping your org chart but more importantly, from the organization’s perspective, you’re missing this great opportunity to really capture those people and keep those people. If you think about this from an outside in perspective, and you think about it from the perspective of that user, this isn’t “I can’t figure out how to use this software.” This feels like a bait and switch. This feels like you told me X was going to be possible, and you have made this impossible. You’ve told me something then given me another. It’s negative brand equity and negative word of mouth. That’s why it’s such a critical issue.

Why the activation problem is never fully solved

Adam: Do you feel that startups ever underestimate the depreciation of particular tactics, like onboarding for example? Because it’s one thing to get it humming, but context is always changing.

Shaun: You’ve straight up hit on one of my favorite topics on this. For example, I look at an activation program as a never-ending stream of work. It’s fundamentally a champion/challenger problem. It’s not, “We will solve it, we will ship it and we will be done.” We ran hundreds of experiments in activation in Atlassian and the vast majority of them failed, but the ones that succeeded did so in such a massive, metric-moving way that, for the investment versus the reward, it was incredible.

There were times when we shipped an onboarding flow, and wondered if it was the best we could do. My argument was always that we’ll never know until we continue mining it. The answer was we always did better. We always managed to do better.

The second reason you clearly must keep doing this, is that your brand promise keeps changing. The features that you deliver to the market change, and if your onboarding is static, then, again, you will fail to meet your promise. Plus, your product itself changes. The way image information architecture exists, it’s just this constant program of work to make certain that you are safely delivering those users from the marketing site into the product, and then from their first touch of the product to getting their value, and from getting their value to forming a habit, and from a habit into high engagement. It’s that simple.

How to get your users to stick around

Adam: I know a lot of organizations are tempted to focus on acquisition, but you really believe that you gotta solve the retention problem first. Why?

Shaun: It never ceases to amaze me how much time we as a industry spend optimizing our acquisition tactics to acquire a huge bunch of people. When you think about all the energy that has gone into this, and understanding who those people are and how to go and find them, and then you look at the number of people that drop off in the first five to ten minutes, the first day, it always breaks my heart. Even at Atlassian, where I felt like we were getting better and better at this, we had the charts that would show us what was happening. Every one of those people is a person whom you have fundamentally burnt. You have failed to give them what you told them you would give them.

The most authentic form of acquisition, by far, is word of mouth.

When I think about the ROI of things that you can do in a business, make certain that your customer is safely handed from acquisition to the activation. Make certain that they are activated and you have done everything in your power in order to make certain they have found their “Aha” moment and they have began habit forming. Then make certain that they’re getting the maximum value from your software though engagement. Those are generally very low investment, but with potentially massive rewards. Not only because you keep those users, but if you truly succeed at that, and you get high engagement, then what you really get is sustainable businesses, because you get word of mouth.

People who are highly engaged with your software are always the people who love it, and those people who love it will tell other people. The most authentic form of acquisition, by far, all day, every day, is word of mouth. It’s amazing the business you can build once you have that engine going.

Adam: What’s your recommendation when it comes to engaging those users who love using your product and evangelizing what they have to say?

Shaun: You really just want to drive them up the engagement curve. You need to have a way of understanding where your users as far as date goes. Are they brand new? Have they gained any value? Have they had that “aha” moment? Has the habit forming loop happened? Have they achieved time engagement?

Most of the magic happens by itself – people who are highly engaged have a tendency to tell their friends. But you can also literally just ask them. It’s like a kind of obvious tactic in the industry now, but if someone gives you a nine or a 10 NPS and then you say, “Hey, thank you for that. Would you mind tweeting about us?” Or, “Would you mind emailing one of your friends right now?” A lot of people not only won’t feel insulted by that request, they’ll say, “That’s actually a great idea. I didn’t even think about that. I really want to tell my friend about it.” They want to help your business. They feel an affinity because you have solved something that they care about, and that’s what I mean about the authentic element of that.

Adam: I know you can’t prescribe tactics without knowing anyone’s business, but where is the low-hanging fruit when it comes to retention strategy?

Shaun: When it comes to retention, again, I would emphasize having an understanding of what your view is at your activation loop. Your problem is almost always going to be in activation first. If you have $100 and you’re starting up, I’d bet $80 of it in the activation phase, because what happens is people fail to have their “aha” moment. And even if they have their “aha” moment, they can drop out before it becomes a habit. There’s a huge amount of value there.

If you have $100 and you’re starting up, I’d bet $80 of it in the activation phase.

The tactics I use in that space are first understanding what my drop off rates are, and then watching users in their very first experience. I try to understand, “What are the buttons they don’t find? What are the things they are confused about?” Then try really simple stuff like a dialogue box saying, “These are things you are probably looking for.” Lead them in the most likely three places they want to go. It’s amazing.

At one point we had a 12 step onboarding flow at Atlassian. It was deeply involved, it had a whole bunch of things that it taught you, and it was very successful. But when I talk about champion/challenger, we were constantly trying to beat that, and we later on ended up with an onboarding flow called “Choose your own adventure”. It was one dialogue box with three buttons, and it outperformed the 12 step program. The 12 program was trying to tell you enough that you could do the rest, but it turns out that most people arriving at the software wanted to do one of three things. Rather than needing to educate them about those three things, about the way in which they could go about thinking about the software and find it, just giving it to them was enough.

Introducing a growth mindset within your organization

Adam: A common challenge is bringing a growth mindset into a startup organization that doesn’t already have a growth team. The idea of embracing things like iterative improvements or being agnostic to process. What are the greater cultural challenges that a startup faces when adapting this mentality?

Shaun: Large businesses or small businesses face similar challenges when they’re introducing a growth team. When you’re focused on trying to solve a bunch of market problems that exist, you get used to shipping big answers to big problems. It’s what you do, and the growth mindset is a little bit antithetical to that. We don’t know exactly what the problems are (in growth), but we can take some guesses, then we can test whether or not we’re right or not, and we can make that a butter churning machine that we can constantly turn over.

There’s this natural anxiousness or misalignment, particularly in startups because there’s competition for resources. When you’re trying to ship really big, hard things, you’re thinking, “Why do you want a team of five people off doing rats and mice work?” The answer is that we are shipping all of this great work and this work deserves to be used. It deserves to be connected to our users. While we are doing big things, it’s very hard for us also to go back and find all those places where it’s not being captured. Our customers deserve that from us. They deserve that we will work out where they are getting lost and being unable to capture the value, and we will help them succeed. Not only will it help our business, but it literally is the right thing to do for our customers.

Adam: How do you make sure that, once you’ve established that mindset, the growth team doesn’t then become siloed? How do you make sure the communication lines are still open and clear with product, design, marketing, sales, support, etc and growth isn’t doing these experiments on an island?

Shaun: That’s when things really get tough in the growth world, when they’re viewed as a potential competitor, or just off doing their own thing. You need to be very careful to make certain that those organizations are your partners, not your competitors. In some cases, you end up with marketing competing with growth, growth competing with product, and everybody competing with everybody else.

We all have a job to do. Marketing has a product to tell people about, product has the job of producing more value that we can capitalize on for our customers, and growth’s job is to make certain that as many people get connected to that value as humanly possible. If you set that up as a framework of roles and responsibilities, and you’re clear about what it is you are there to do, which is extract that value and help and work with other teams, you tend to get much better results.

When you’re first building your growth team, there’s an early moment where you need to get some wins. That can be competitive, and it can feel like you’re poking holes in other people’s software. Don’t look at that as product shipped something wrong, because we came in and we changed two buttons, and suddenly we got 12% better activation. That’s not the principle. We need to be clear that not what we’re doing. We’re not doing this to prove that you’re wrong. We’re not doing this to prove that your processes weren’t correct. We’re doing this because it’s the nature of the world that we need somebody to come through and apply this thinking.

Adam: Before we go, where can our listener’s go to find out more about Metromile or your thinking on growth in general?

Shaun: If you want to learn more about Metromile visit metromile.com. We’re presently in seven states in the US, and we’re transforming insurance. In terms of my thinking about growth, I’m not really a big blogger, but I can thoroughly recommend the Reforge program, which Brian Balfour runs. He’s a really sophisticated thinker in this area, and I think it’s a great place to start if you’re looking to get deeper in growth.

Adam: I hear you’ve been known to speak there.

Shaun: Once or twice. If he can twist my arm.

Adam: Shaun, this has been fun. Thanks again.

Shaun: Thank you very much. Cheers.