Ryan Steinberg on fostering empathetic automation

In our quest to make the most of support automation, we’ve found three uses that really optimize its potential.

The first is routing: directing a customer’s question to the right team. Next is assisting: gathering information from a customer so a human support agent has everything they need by the time they’re stepping in to help. Finally, there’s problem-solving: answering questions faster and more effectively than human agents can.

But if there are ideal use cases, there are also stumbling blocks to avoid. You wouldn’t want to deploy automation for a customer who has specifically requested a human touch, for example.

Today’s customer support teams must walk a fine line between robotic troubleshooting and empathetic help. It’s all part of keeping support personal, one of Intercom’s primary missions.

In this episode of Inside Intercom we sat down with Ryan Steinberg, Intercom’s Head of Global Support Operations, to discuss just that. Their conversation touched on everything from the best and worst applications for automation to the future of support operations.

Short on time? Here are five quick takeaways:

1. We’ve developed a new metric called ROAR: the rate of automated resolution. It tracks end-to-end resolutions of customer questions (without a human ever stepping in), and it’s saved us $400,000 in the past year.
2. Is faster better? As long as you’re setting clear expectations and delivering quickly once the conversation has begun, customers don’t mind waiting a day for a first response.
3. Your support approach must take into account where the customer is in their journey. Even a lower-spend customer represents a huge opportunity to build loyalty and revenue by solving big problems quickly.
4. Though many see automation as impersonal, it’s actually the opposite: it fundamentally requires your team to think deeply about the customer’s journey and potential roadblocks as you aim to respect their time and sanity.
5. There are three applications that require caution when working with automation: sensitive topics, customers who have explicitly asked for a more human experience, and new customers whose faith in you may be delicate.

If you enjoy the conversation, check out more episodes of our podcast. You can subscribe on iTunes, stream on Spotify or grab the RSS feed in your player of choice. What follows is a lightly edited transcript of the episode.

Kaitlin: Ryan, welcome to Inside Intercom. We’re so happy to have you here today to talk about how you and your team have helped us scale Intercom’s support experience with automation. Before we dive into that, can you give a really quick background on your role here at Intercom and how we work together?

Ryan: Sure thing, Kaitlin. Thank you for having me. I’ve been with Intercom for a little over four and a half years now. I started off as a front-line customer support representative talking to our customers. After about a year, I moved into an operations role, something I was super comfortable with. I studied econometrics in college, so I was very much into the numbers game, and I’ve been doing that for about three and a half years now.

“It’s been a hell of a run, and I’m very excited to still be here and excited to talk to you today”

The start of that role was that we were relying on Intercom reporting, which wasn’t all that robust. We needed people to go into our databases and start writing SQL queries to pump out more information so we could really up our game as an operation. Right now, we have a team of people around the world. And now we’re covering everything on the operation side of things, reporting on individual metrics, team metrics, headcount, planning, forecasting – all that fun stuff. It’s been a hell of a run, and I’m very excited to still be here and excited to talk to you today.

The challenge of automation

Kaitlin: Brilliant. As a support director, I’m sure I’m not alone in saying that we could absolutely not live without our support operations team. And it’s been so wonderful to see not only you and your role continue to grow, but the team you’re building around it and the huge impact it’s had. Focusing in on automation, optimizing the support experience for a company like Intercom that’s scaling so fast is a challenge. Can you share some of the challenges that we face broadly speaking?

Ryan: When I joined the Intercom support team, we had 14 people on the team spread out across two different offices in Dublin and San Francisco. We ramped up pretty quickly as Intercom added boatloads of customers over the years, and we actually added two more offices in Sydney and Chicago and got the team up to more than 100 people at one point. Scaling along the way, there are a ton of challenges, probably the biggest of which is just how the hell do you communicate to 7 times the number of people you were talking to previously? There’s a lot of that: relying on Slack, relying on Intercom within the team communication technologies we have.

“A lot of that has to do with automation, and a lot of that has to do with the tiny efficiency wins you can find throughout the operation”

But then, as we increased the operation, we were segmenting our customers and segmenting internally. There were all these different operational nuances we needed to really dig into and hone in on. The biggest problem we faced as an operation was that, for the longest time, we were basically adding customer support heads in lockstep with our customer growth. So, for every X number of customers that we added, we were adding another person on the CS team. One of the big things you and I focused on over the past two years, has been trying to decouple the rate at which we’re adding heads with the rate in which we’re adding customers. A lot of that has to do with automation, and a lot of that has to do with the tiny efficiency wins you can find throughout the operation. We’re obviously focusing more on the former today.

Kaitlin: Absolutely. You recently wrote a blog post that talked about how automation helped save Intercom $400,000, which is not nothing. This really allowed our support team to increase our efficiency at scale, so can you talk to us about the automation strategies we’ve used and how they’ve helped us to achieve these exciting outcomes?

Ryan: For a long time, the only bit of real customer-facing automation we had was this thing called “article suggestions”, which was a bit of machine learning that basically read through what your customer was writing to you about. It read through all of the different help center articles we have and tried to suggest ones that might be appropriate for the customer in hopes of resolving that conversation without a human. It resolved about 0.5% to 1% of all of our conversations.

“I love acronyms. I have some beef with Elon Musk about it”

But in August of 2018, we released this thing called Resolution Bot, which basically allows you as a support team to take some of your most commonly asked questions and create boilerplate responses. Then, using a very similar machine learning model, it read through all of the different things your customers were saying to you and tried to match those pre-written responses to what your customers were asking you about – not just throwing articles at them, but also proactively trying to answer those common questions. Things like: “How do I change my email?” or “Where do I change my profile picture?” It’s really good at solving the simple easy transactional, “How do I…?” type of questions.

From there, we bucketed these two things together into a metric called the ROAR, which stands for the “rate of automated resolution”. I love acronyms. I have some beef with Elon Musk about it. You can talk to him about that later if you want to. But ROAR basically looks at all the conversations that Support was going to have a hand in and what percent of those are being resolved automatically without a human ever stepping in. This is end-to-end resolution of customer questions. And I’m happy to report that with Resolution Bot and a bunch of really cool updates we made recently, we are between 4% and 5% week to week. If you’re doing the math, that basically translates to around four heads on our team. All of these numbers are approximate, but it’s around four heads that we didn’t have to hire. If you’re using the most conservative, fully loaded cost of an employee, it’s around $400,000, $100,000 per head. That is money we saved last year when we were at 4%, and it continues on every single year, so that’s $400,000 that we’ll save this year, next year and to infinity and beyond, I guess.

How to leverage Resolution Bot

Kaitlin: What’s also interesting is that we were pretty skeptical that Resolution Bot was going to be able to have a meaningful impact on our customer experience, our team and our ability to scale. Because the breadth and depth of our product offerings and the variety of customers using our products is just so vast that perhaps Resolution Bot would be better suited for a business with a narrower focus and thus a narrower set of customer questions. But even at 4%, it was really exciting for us to have been so pleasantly surprised at the meaningful impact that this could have. How did we leverage this tool internally, and how might some of Intercom’s Resolution Bot users do the same?

Ryan: The big lever we have is our product team, obviously. They’re constantly working on this and tweaking the models to make them better. But internally in terms of how we operationalize these things, we did two things. The first one was that we had all of these different individuals around the world: different managers who are individual contributors who know the product and our customers’ questions really well. The first thing we did after we released Resolution Bot was we had these individual contributors work in a functional team to create new answers, update old answers and work with our marketing and product teams to have all the right information to answer people’s questions automatically. They’ve been a big help.

“Focus breeds excellence”

More recently, we actually created a role on my team, this wonderful ex-support person at Intercom. He is really focusing on all of our automation technologies, heading up that special team and helping people get the information they need to tweak old answers, create new ones to resolve customer conversations. He’s doing that in addition to working with the product team and saying, “These are the pieces of information I think would be beneficial for me and our customers to have to improve things.” He’s been really digging in there and exemplifying what our VP of Sales and Support, L.B. Harvey, likes to say: “Focus breeds excellence.” He’s really digging in there every single day and seeing some pretty incredible results. We’ve seen about a 1% lift since he joined my team back in November.

Kaitlin: It’s great. I was actually going to use the same quote. It’s wonderful. Focus certainly has bred excellence for us here, but really we’re just getting started. I can’t wait to circle back in six, 12 or 24 months from now and see – with one dedicated, experienced person focused in this area – where we can continue to drive wins.

Ryan: Yeah, and we’re very confident about that. Both Garrett and myself have been talking with the product team and seeing the opportunities we have ahead. We’re feeling very good that we can get that number up from 4% to 7% by the end of the year. And we’re making very good progress in the beginning of this year so far. So fingers crossed, but I’m very hopeful.

Is faster better?

Kaitlin: As you and I both know, it’s generally believed that faster is better when it comes to support specifically around response times. I think we know that, as people who live and breathe support here at Intercom but also as consumers and customers ourselves. So response time is very important across the board, but certainly when looking at metrics like customer satisfaction (CSAT) and the impact it can have on business metrics like retention, conversion, activation, etc. We took a closer look at this last year and challenged this notion of “faster is better” in every situation all of the time, given our need to scale and that linear growth of both our customer base and our headcount. Can you talk to us a little bit about how we approach challenging this notion and the impact it has had on our business?

“At the end of the day, expectation setting is the name of the game here”

Ryan: Definitely. A couple of years ago, I ran this big analysis. It was over a hundred thousand conversations at that point, looking at what impact the various time-based metrics actually have on the customer experience, using CSAT as a proxy. And what we found was this: among the first response times, the subsequent response time (how quickly we’re getting back to a customer after that first response) and time to close, the two metrics that really mattered the most were subsequent response time and time to close. After doing a bit more research, we realized that at the end of the day, expectation setting is the name of the game here.

So long as you’re properly setting expectations with your customers around how long they need to wait to get their support question first answered – and as long as the conversation moves pretty quickly after that point – customers actually don’t mind waiting as long for a first response.

We saw that research and wanted to test this out with some of our customers. So we took about 20,000 of our customers and chopped them up into two groups. One was going to get the normal support experience, which is around a two- to three-hour first response time. And the second group was going to get a one business day experience, so typically around 24 hours. We ran this test for about six months and had many tens of thousands of conversations with these customers. And what we found was very interesting and very much in line with the research that we previously found. When it came to the two- to three-hour first response time versus the one-business-day first response time, CSAT was only 1% lower.

It was a difference of 96.5% happy to 95.5% happy – which is a 1% drop – but we were comfortable with it so long as the bigger picture business metrics held up. And I’m happy to report that when we looked at customer retention between the two groups, there was a less than 5% difference in the two. And when it came to net revenue retention, there was actually no difference between the two groups, which was a phenomenal result. So with that, we’ve rolled this out to some of our lower-spend customers and we’re continuing to experiment with looking at when customers actually need fast support versus when we can give them a little bit slower of a first response time.

“What we really want to do in the future is take information about where they are in the customer journey, but also what are they actually asking about?”

Kaitlin: You talked about spend there, and I think it’s helpful to share that in parallel with this automation strategy. We’ve been talking a lot about segmentation: both segmentation of our customer base and segmentation of our teams. But we’ve also talked about where in the life cycle the customer is. To your point, are they in need of activation? Is this a newer customer? So that’s a whole separate podcast episode, but interesting to think about in line with the automation conversation.

Ryan: Yeah, and we’re very excited about the automation potential here too. With that experiment, the only thing we were chopping people up by is how much are they spending and that’s basically it. What we really want to do in the future is take information about where they are in the customer journey, but also more importantly, what are they actually asking about? Because at the end of the day, for someone who is paying us hundreds of thousands of dollars a year, if all they’re writing in about is a simple feature request, we probably don’t need to give them a super speedy first response, so long as expectations are set there.

And on the other side of the spectrum, if somebody is on our early stage program and really only paying us $50 a month, if they’re telling us that the API is down or that there’s a P1 issue, we probably want to get back to them pretty quickly despite the fact that they aren’t paying us all that much, because that issue in that moment is really impactful and really important to them. So finding that line between customer attributes and conversation attributes is the next stage of this experiment.

Keeping support personal

Kaitlin: One thing you and I both know is that Intercom’s mission is about keeping our support personal. We’re thinking about how and what that looks like in support even as we scale, and we introduce things thought historically to be impersonal, such as automation and chat bots. Do you think automation can ever be personal? And what does being personal even mean today when it comes to this ever-evolving space of customer service?

Ryan: The amount Intercom has evolved over the years on this particular “What is personal support?” question is pretty fascinating. When I first joined, we were taught that personal support means that you are talking to a human at the other end who respects your time. But you could have a good time, whether you were sending GIFs or emojis back and forth. You could build a real one-to-one personal relationship, whether that’s over time or just in that simple interaction that you have around a support question. But over the years, what we’ve realized with this rise of automation is that we are in the “trough of disillusionment” when it comes to chatbots.

“Automation is actually a super personal way to provide support”

We’ve realized over the years that automation is actually a super personal way to provide support, because it means you are thinking about the time your customers have to spend trying to get answers to the questions they have about your product. If a piece of support automation can step in and answer somebody’s question in 30 seconds – 10 times faster than any human could possibly do it – they didn’t build any personal connection with an individual on your support team, but you respected their time as an individual. They got their question answered, they moved on with their day and continued to use your product in a more effective manner than if they had talked to a human there.

So there were these particular situations in which there’s a simple question or a simple ask in which support automation can actually provide a far more personal experience by respecting somebody’s time more than a human possibly could.

Kaitlin: What advice do you have for companies like Intercom that want to build an automated support experience that pays off for their team and their business but also their customers?

“Support automation, via that metric ROAR that I mentioned, can actually solve questions faster and more effectively than any human possibly could”

Ryan: There are about three categories that support automation can be good at. The first one is really simple: it’s routing. Somebody writes in using keywords X, Y and Z, and you need to route those to team A. Automation is really good at that. It can read the text and can pass it along very effectively, far quicker than any human possibly could. So that’s really good and that’s been around for decades now.

Ryan: The second one is assisting. This comes back to that expectation-setting piece that’s on the customer end, but there’s also internally letting people know information about a customer from the onset. If somebody is writing in about a problem, and you know you need information about it, a bot can be really good at asking those questions and putting your support reps – the people who are actually talking to these customers after the bot is finished – in the best position to solve this thing quickly. When your customer actually gets in touch with the human, they’re providing answers to the questions that you’ve already asked via this bot.

“It’s important to be compassionate with what a person is actually going through”

Ryan: And then finally is this problem-solving piece. Support automation, via that metric ROAR that I mentioned, can actually solve questions faster and more effectively than any human possibly could. With a team of our size, where we have 80 people around the world talking to our customers, if we have a new feature, and we update an old bit of our product, we need to update that information in 80 people’s heads. And we’d probably have to say it six or seven times, as the old saying goes. With support automation, all you need to do is update it once, and it can send that out in a distributed manner to all of your customers. So that’s really, really competitive.

When to avoid automation

Kaitlin: What should be avoided in this space? When should a chat bot or automation not be implemented?

Ryan: Jeez, that’s a hard question. It really depends on your business at the end of the day. If you’re a freemium model when it comes to your customers, you probably want to delight them a little bit more – despite the fact they are free – to try to get them into that paying experience. But in general, what we’re seeing with our customers and internally is that there are roughly three different areas in which it’s good to be careful around using automation. The first one is sensitive topics. If somebody has like a billing P1 where you overcharge them by 10X or whatever it might be, you probably don’t want to make them go through an extensive automation tree before they get in touch with a human. It’s important to be compassionate with what a person is actually going through.

Ryan: The second one is high-touch customers. We have customers who tell us specifically: “We do not like this automation thing. We hate it. It makes for a miserable experience. We’re fine waiting longer for a resolution if we can talk to a human.” If people are specifically telling you they don’t like the way this feels, you should try to improve it. But at the end of the day, it’s probably best to respect their wishes, if they are fine with the negative consequences or loss of efficiency on your team if you’re not using that automation. Then lastly, it’s that new customer bucket I mentioned. So if you have a premium model, you need to make sure that you’re providing a good support experience to those free customers. Or even if you have a hundred customers as opposed to over 25,000 like we have, it’s probably best to try to delight each and every single one of them, because the scale is just so much smaller.

Kaitlin: Sure. I think on the other end of the spectrum is white-glove service: your VIP customers, however you define them. So Ryan, as the old saying goes, “What gets measured gets managed.” Measurement is just a big part of your team and our world. How do you go about measuring the impact of the automation?

“If people are specifically telling you they don’t like the way this feels… it’s probably best to respect their wishes”

Ryan: Within ROAR, we actually have a step lower that we look at, which is two different metrics that boil up into ROAR. The first one is coverage rates: of the conversations our support team was going to handle, what percent of those is automation given the opportunity to step into? Maybe that’s sending an article, suggestion or an answer to try to resolve that conversation. Then, when a conversation does get a piece of automation trying to suggest a solution, what percent of those are actually being resolved? So that’s the resolution rate and if you multiply those two together, you get ROAR. But those are the deeper-level metrics that we look at.

Ryan: On the other side of things, we have this thing called the Inbound Custom Bot. When somebody opens up the Intercom messenger and tries to start a conversation, you can give them little buttons and a logic tree to collect information, route things appropriately and so forth. We’re testing this out in the UK and Ireland right now, and there are four key metrics that we’re looking at here.

Ryan: The first one is engagement rate. How often are people actually interacting with this logic tree? Second one completion rate. How often if they engage with the logic tree, and how often are they actually getting through it? Then the third and fourth ones are around qualitative feedback. When talking to our customers who have interacted with this flow a couple of times, what did they think? Internally, we’re asking people for qualitative feedback as well, because at the end of the day, if we’re routing things more appropriately, and we’re getting more information for our agents and our team, we want to make sure that it’s actually helping them in some way. That’s qualitative feedback there. Then there’s obviously quantitative spin on that, looking at time to close. Is this actually helping us resolve conversations a little bit faster, because we have the information from the onset to solve the question?

The future of support ops

Kaitlin: Looking ahead in the not-so-distant future, support ops is an area that more and more businesses are investing in. I’ve previously chatted about this with other support leaders on insight Intercom. What is the big bet that you would make for the future of support ops? This is a growing industry, so to speak. We’re seeing more roles pop up here. What do you think is coming down the line?

Ryan: The first thing we’re already seeing is that more and more businesses are going to get a support operations team earlier on in their journey as a business, because the value there is just starting to become so much clearer, particularly in this automation space. If you think about Garrett’s job in managing the Resolution Bot or Inbound Custom Bot, with the rise of automation, he’s basically taking on the job of multiple different individual contributors or managers. In that world, I think businesses are going to see a very clear benefit in that they can hire this higher-paid, higher-value employee who can do the job of multiple support people at scale via automation. So that’s really compelling.

“We’re already starting to look at how people are actually using their products and drawing direct lines to how they interact with your support team”

I’d say the second thing is that I think there’s going to be a lot beefier analytics going on in the support operations space. We’re already starting to dig into this a little bit, but starting to look at how people are actually using their products and drawing direct lines to how they interact with your support team. What are people doing in your product that leads to the most questions? You have to build that data bridge between your support team and your product team. I think that’s going to be a super big thing that’s coming up in the future.

Kaitlin: Great leaders and great companies not only drive excellent productivity and output from their teams, but they also really invest in skill development, and they hope to see folks on these teams go on to new roles and new teams and have a nice, long run at companies. Thinking about a career in support ops, what are the key skills a leader like myself should think about in terms of investing in a support ops team? If you’re building your first support ops team or you’re growing it, what kind of skills are you looking for there?

Ryan: I’m definitely biased here, but I think that having people who have done the support job previously provides them a level of insight and empathy at the end of the day that allows them to do the job really well, because when you’re doing operations it’s really easy to just get sucked into the numbers and the processes and the blank Google Docs you’re making things out of. Having somebody who really can connect personally to the people that you’re actually impacting – whether that’s the agents on your support team or your customers – is super important.

“The rise of the support operations team really allows you to hire great managers onto your team to really focus on skill development and engagement of your teammates”

A front-line customer support job is a really kick-ass way to learn those skills. More specifically around like the number side of things. SQL is a big must, just because a lot of the reporting out there is just not fleshed out enough for you to do things out of the box. So you need to dig into your own. Then you want basic statistical analysis skills. You don’t need to be running t-tests most of the time, but being able to talk about statistics with people is super, super useful.

Kaitlin: I’d say there’s probably also a dash of Excel or Google Sheets wizardry in there, as well.

Ryan: Definitely. I have one question for you, Kaitlin.

Kaitlin: Yeah, hit me.

Ryan: On the support ops side of things, what are the things you’re seeing? What gets you excited about partnering with a support operations team, and where do you think the industry is going at the director level?

Kaitlin: One big thing comes to mind. Last year, we had a big message for our front-line managers, and that was “People, not projects.” Here we are a year or two later, and we still have our amazing managers not only leading productive teams but high-impact, global projects. The rise of the support operations team really allows you to hire great managers onto your team to really focus on skill development and engagement of your teammates or your agents – whatever terminology you want to use. There’s a long list here, but that’s really the first one that springs to mind: don’t stretch your front-line leadership across too many things.

Then if it wasn’t clear here in our conversation today, I’m delighted with the progress we’ve been able to make in the automation space. We made up an acronym for a metric, and now it’s ubiquitous across the company as it continues to climb. We’re more and more excited. I think we originally wanted to call it ARR for “Automated Resolution Rate”, but clearly that was going to cause some confusion. So I’m delighted that we landed on ROAR. I’m delighted with the automation gains we’re making, as well as the focus it gives our management teams.

Kaitlin: With that, Ryan, we will wrap up. I want to say a big, giant “thank you” to you and to all the great support ops leaders out there. I don’t know where we would be without you. I’m excited for International Support Operations Appreciation Day to launch, coming to a support team near you. Thank you so, so much for your time and the great partnership.

Ryan: Thanks for supporting us through and through, and thanks for having me today.