Intercom's Karen Church on product analytics

How are customers using the product? Are they finding value in it? What story do those insights tell us about what the product team should do next? At the heart of this analysis is our Product Analytics team, and to get a better feel for how they think and operate, I hosted Senior Manager of Product Analytics Karen Church on our podcast.

Karen came from Yahoo to Intercom in January 2016 to create a cross-functional team of data analysts, data scientists and data engineers. Together they generate insights to inform the design, development and measurement of Intercom products.

In our chat, Karen covers the product metrics that matter most to her team, why storytelling skills are crucial in any data-driven job, how analytics teams can better integrate their product counterparts, and much more. If you enjoy the conversation check out more episodes of our podcast. You can subscribe on iTunes or grab the RSS feed in your player of choice. Below is a lightly edited transcript of the conversation. Short on time? Here are five quick takeaways:

1. Before your startup brings aboard its first analytics hires, and to set those hires up for success, you need to get the underlying data infrastructure, pipelines and engineering in place.
2. Karen’s team is focused on three key metrics: intent to use, activation and engagement. They’re also beginning to look closely at customer success metrics.
3. An insight that isn’t acted upon is essentially worthless, so Karen and her team allocate 40-50% of their time to crafting stories around the data they’re surfacing.
4. One simple way an analyst can get closer to the product teams they support is to join their daily standups. It gives each side regular context into the other’s work.
5. An analytics team should empower every employee to make data-informed decisions. Karen’s team does this by regularly sharing insights company-wide via Slack and offering tutorials for those who want to unearth insights themselves.

Geoffrey: Thanks a million for joining us, Karen. To get started, could you give us a feel for your career before Intercom and what drew you to analytics?

Karen: My background is computer science. I did an undergrad at Institute of Technology Blanchardstown in Dublin, which is where I fell in love with programming. I loved it so much that I did a PhD at University College Dublin. The focus of that was trying to understand and explore how people searched and browsed the web from their phones back in like 2004, 2005, pre-iPhone. I knew I didn’t want to stay in academia, so after the PhD I moved into industrial research labs where the focus is really on innovation, which I found really exciting.

I moved to Barcelona and spent five years at Telefonica Research, exploring future mobile information access, trends and behaviors. Things like mobile messaging and mobile search, mobile app recommendations and human-computer interaction. I loved that. Then an opportunity came up to move to the Bay area and lead a team of scientists at Yahoo Labs. That team was focused on native and mobile ad analytics. We explored patterns and behaviors in various mobile data sets to improve advertising experiences, which sounds like the dark side. But it was really interesting. I had a great team there. And then about 18 months ago, I joined Intercom to lead the product analytics team.

On the data piece, I’m fascinated – maybe borderline obsessed – with people, how they behave and how they interact with technology. Data enables me to investigate and understand that. It’s real behavior, and I find that really interesting. I’m also very nosy as a person, so that probably helps a little bit too.

The makings of a product analytics team

Geoffrey: How would you describe the mission and the makeup of the product analytics team at Intercom?

Karen: We are a growing team of data scientists, product analysts and data engineers, and our mission is to help Intercom make more informed, effective decisions using data. We partner, in particular, with the product teams.

There’s the data engineering side where the focus is on designing and building and scaling our infrastructure, our pipelines and our data sets so that we can ultimately democratize that across the company and enable anyone, even beyond the analytics team, to access data to make decisions.

Then on the analytics and data science side, it’s about analyzing, exploring and mining patterns and behaviors and trends, and surfacing those patterns in a meaningful way that either guides a key decision or informs a new product.

Geoffrey: Were there many analysts here when you arrived at Intercom?

Karen: Kevin McNally was the very first product analyst, and he’s now a senior data scientist on the team. He’s been here almost three years. Then I was hired to lead the team, then Nick Odlum, who was our second analyst, and now there are seven of us. We have two on the data engineering side and four on the analytics and data science side, and then I lead the team. We also have a couple of amazing engineers at the moment, doing tours with us, which is really cool. Folks across R&D and Intercom are really excited by data, so they’re actually coming to spend some time working with us too. That’s been cool. We have two at the moment.

The role of analytics at a startup

Geoffrey: When does a startup need to bring a product analytics team into the picture? Is there a certain point in the company’s life that signals it’s time to start building a team?

Karen: I don’t know whether there’s a certain point in time, but I would bias towards earlier is better. Even before thinking about the analytics piece, the data piece is much more important. The underlying data infrastructure, the pipelines, that engineering piece. An analytics team is only effective if they can access and analyze data to surface really compelling insights. You need the underlying data stores and infrastructure to be operational and efficient in order to do that. We were a little late to the game there, but we’re investing really heavily in it now.

Hire on
the data engineering and infrastructure side first.

I would hire on the data engineering and infrastructure side first. And often the first person of this kind in any startup will probably have a mix of (experience in) data engineering and data wrangling and data analysis, and that’s good. Maybe you can find someone who is passionate about the underlying data stores as well as analyzing and surfacing compelling insights from that. Hire a good data person first and then you can think about how to scale that team.

In terms of product analytics as a function, as long as you’re building a product that’s being launched to market, you should be making sure that product is tracked correctly, that you have some way of gaining insights into how people are actually using it, and then you can use those insights to iterate on that product or inform a new one. But I would bias towards early.

Geoffrey: What are the challenges of retrospectively adding a product analytics team a couple of years into a company’s life?

Karen: That’s an interesting question. It’s not necessarily in the DNA or makeup automatically. If you’re startup is founded and led with a design direction or a more qualitative user experience research direction, it’s in the DNA. People understand it, they get it. When analytics is brought in a little late it just means that it’s perhaps a little less understood and there needs to be more work in terms of helping people understand it, pivoting that culture and building the reputation of that team.

We’ve done some great work over the last year and a half to help there, and now I would say that lots of teams across Intercom are pretty data savvy. They’re pinging us left, right, and center to work us and help them, even beyond product, which is a great testament to some of things we’ve been doing.

Geoffrey: So it’s a culture shift and organizational problem as well.

Karen: A culture shift, definitely. I came from very, very, very data-driven companies. Many of Yahoo’s products are very, very data-driven, data-informed. And when I moved to Intercom it was quite different. We think very differently. We care greatly about offering a really personal experience to our customers, and data just wasn’t the initial light bulb moment for that. More and more, we’re seeing that you can absolutely build amazing, personal customer service experiences while using some data to help you gain insights into how those things are being used, as well as help guide smarter automation within those products too.

The product metrics that matter most

Geoffrey: What are the key product success metrics that Intercom uses today, and why have we chosen them specifically?

Karen: We approach this, at least in my experience, a little bit differently than other companies. Google has the HEART framework (created by Senior User Experience Researcher Kerry Rodden), which helps you think about your user’s experience of a product and the metrics that will matter most to measuring the success of that product.

We partnered really closely with our product teams to understand what they’re building, why they’re building it and what problems a particular product or feature is trying to solve. The result is three metrics that we talk and think about a lot at Intercom.

The first is intent to use. This is the action or set of actions that show that your customers are deliberately intending to use this feature or this product. This is important because not all features you build are meant for your entire customer base. You might actually build a set of features that are meant for a subset of your customers.

Not all features you build are meant for your entire customer base.

The second one is activation. This is about the point in time when a customer first realizes the value of a product. Again, it’s the action or set of actions that indicate that realization of value for the first time. If your customers don’t realize the value of your product, they are never going to go on to be fully engaged, happy, successful customers who keep returning for more and more of that experience. And that’s what you want.

The third one is engagement. It’s this stickiness piece. The measurement of the continuing return of your customers. Day-to-day, week-on-week, where they’re continuing to realize the value of your product. This is how often are they using features, the time at which they’re using those features, and the overall stickiness of your products and features.

Those are the three. We’re also trying to think more holistically about customer success. Because engagement is just one piece, but what we really, really care about is whether we’re helping our customers be as successful as they can be. We’re doing some work around that at the moment.

The importance of telling a story with data

Geoffrey: For your team to be successful, the product teams ultimately need to use the data you’re providing them to inform and guide their decisions. What role does storytelling play in your team’s work?

Karen: It plays a huge role. We talk a lot as a team about carving out 40-50% of our time just crafting the stories that we’re going to tell around the insights that we’re surfacing. Anyone whose background is in analytics or data science has produced what they think is a great piece of work – where they’ve surfaced a really exciting insight and they share some report or a deck or chart – and it lands flat on its face. The insights we surface are really only as good as the story we tell. Any insights that aren’t acted upon or used, they’re worthless.

So we spend a lot of time crafting good stories, and we do this in a number of ways. We have regular feedback sessions a couple of times a week as a team, where we actually share insights that might be in flight. It might even be a method or analysis that’s happening. And the whole team provides feedback, both on the approach as well as the story and the delivery of the insights. We share informally and provide feedback on Slack too. We give demos and presentations to the whole company, and we always dry run those, too, to make sure that the whole team has a chance to provide feedback on the story or narrative that’s being shared. It’s actually one of the underrepresented skills a great data analyst or data scientist needs.

Geoffrey: Thinking back to your days in university, was that something that was taught?

Karen: I was really lucky. My PhD advisor is an amazing data storyteller, and he really worked with me, as he did with all of his students, to help me shape a narrative when we published academic papers. When you publish a paper, you would then present at an academic conference where you’d have to stand up and tell everyone about your piece of work. He worked with us on those things. But it’s something that you should continually improve, and I have plenty to learn and improve from.

The insights we surface are really only as good as the story we tell.

Telefonica took us on this two-day course where they got this wonderful guy to come in. He used to be involved in the Gaiety Theatre, and he did this two-day intensive workshop on storytelling and presenting. It was life changing. It really, really was. We had to ad lib a lot, and then we’d take a piece of work, actually craft a story and share with the team. That really leveled up how I was presenting and articulating some of my own insights.

The more you evolve and mature in your role, the way in which you tell stories, the visualizations that you use, they all mature and evolve with you. I actually gave a talk recently and I was including some work I did as part of my PhD. I looked at my thesis and thought, “That is terrible. I cannot believe that was the quality of the work.” I was presenting some really interesting insights, but they were in this black and white table; there weren’t any visuals whatsoever. People can interpret a lot through visuals, so I think that visual component to storytelling is really important, too.

Getting closer with your startup’s data

Geoffrey: What’s the one piece of advance you would give to a PM or a designer or an engineer to make sure that they’re getting the most out of their startups’ data?

Karen: If there’s an analytics team there, go and talk to them. We use Slack a lot within Intercom. We set up a Slack channel called #ask-product-analytics, where we invite the whole company, not just product, to come and ask us any data related questions. That’s been really successful. If you have an analytics team, go knock on their door. If there isn’t a Slack channel or an email or some website where you can go and interact, then find those people and just go and talk to them about your specific area, the questions that you would like to answer, or even just a general things like what are they types of things I can do with information. I guarantee if there’s an analytics team there, they’ll be crying out to partner more with designers, researchers and PMs.

If there isn’t an analytics team, I encourage PMs, designers and researchers to dig yourself. A lot of our folks here do that. They work with actionable dashboards. They run queries of their own. Our entire engineering org went and did SQL training recently. There are ways to find and query data yourself so you can surface insights. But first port of call is finding the folks that work with the data and just talking to them.

How we structure the Intercom product analytics team

Geoffrey: When companies start to build out their analytics team, they tend to organize themselves in either a centralized or embedded model. You’ve tried to follow a more hybrid approach at Intercom. How does that work?

Karen: We have a single product analytics team, and we all sit together in one pod and we do work together a lot. But we embed as needed. One of the smallest things we did was actually attend the standups for the product teams that we support. That completely changed the dynamic, because the analysts had absolute context for the various decisions and activities that were happening within the product teams. And likewise, the product teams had way more context into the daily focus and work of the analyst in question.

Before, we didn’t really partner at that level. We didn’t attend standups, we didn’t attend their weekly planning meetings. There weren’t really any regular metric reviews. Initializing some of those key meetings, like standups, attending weekly planning meetings, having more regular metrics review sessions or sessions where we explicitly review insights, those small tweaks have really helped. It’s about daily, iterative sharing of insights and context, and all those things have helped build much stronger partnerships. We really need that in order to make sure that we’re actioning on the insights that are being surfaced.

The importance of democratizing data

Geoffrey: I’ve always loved your idea of data democratization. In a nutshell, that’s making sure everyone can access and understand the data that you guys are working on. Is there anything that’s been particularly successful for you in doing that?

Karen: There are a few things. A few months ago we were trying to improve how we instrument our products, and this involved a lot of partnering with great engineers in Intercom to help track the events that we know are important and would tell us how people are actually using these features. But there wasn’t a common language – a shared understanding of what we should be tracking and what we should call these things. It was a bit inconsistent, and it meant that not only was tracking hard, but the resulting analysis that happened was really difficult.

So we generated a new naming convention for events. We’ve been partnering with the great engineers here on more automated tracking, and that has led to shared understanding and a common language. It means our products are much better instrumented, and the resulting analysis is way easier to conduct.

We’ve also been trying tutorials for some of our tooling. Looker is our core BI tool, and we ran some tutorials last summer to help other people in the company understand how to use and access Looker and various dashboards. We absolutely want to invest more in the next six months toward educating folks, because we hear from an awful lot of people saying “Hey, I’d love to answer this question.” They want to be able to do that themselves, and we should be empowering more people. There are lots of great companies who we look towards who are inspiration on this vein. Airbnb is one of them. They have an amazing data school. It’d be really cool to have something like that here at Intercom.

Airbnb’s Data University empowers every employee to make data-informed decisions.

A few months ago we introduced an initiative called Chart of the Week. It was led by Flora Devlin, who’s a senior analyst on the team. Every Friday we share one single chart with a caption, that is hopefully compelling, with the whole company on Slack. The idea here is to try and spark curiosity around some of the insights that we’re surfacing, hopefully get some questions and spark general discussions among teams. There’s been some great reactions in Slack, and also from offline conversations with people. We’ve done it every week since the end of March, and we have a singular page where people can go and see previous charts.

Sharing weekly charts like the one above help spark company-wide conversations around data.

Geoffrey: Talk me through the data school. That’s for any employee to attend?

Karen: Yeah. If we think about Intercom and how we onboard people, we have our own version. It’s an entry school where we teach new hires about various aspects of our business, our history, our mission, our vision, our products, the way in which we work and the way we think. I would love us to have a data portion of that. Maybe it’s a session or a half day where we actually direct people to the various analytics teams they can work with. We set them up with various tools and accounts, we point them in the direction of some tutorials, or maybe give some hands-on tutorials to let people know there is a wealth of data here that will enable you to do your job more efficiently, and here’s how you can access and consume it in a way that will help you in the long run.

Striking a balance between data and research

Geoffrey: One thing that’s talked about at a lot of startups is this idea of being data-informed over data-driven. How does your team strike that balance between getting the most out of the data available to you, but appreciating that data might not always give you the full picture?

Karen: It’s definitely a balance. I actually love the phrase evidence-based rather than those two. Even as analysts and data scientists we know that quantitative data is just one piece of the picture. We partner really closely with our qualitative research team here. Analysts and researchers are partnering at the beginning of projects and they’re asking questions like, “What do we want to learn here, and which method is the best at learning?” It’s not a case of data versus research. It’s a case of whether one method better in this instance. And in most cases both methods are really good at providing a much more holistic picture of our user’s experience.

I’m a big believer in combining qualitative and quantitative insights, and it definitely gives us a much more rounded picture. Even when we’re moving more towards advanced analytics, like predictive analytics or applied machine learning where we’re actually trying to inform and drive some of what we’re building using data, we always ask, “Does this problem require us throwing machine learning at it or is there a simpler solution that will actually solve our customer’s problems without this?” As long as we keep asking those things, we’ll be in a good place.

Geoffrey: Does your team ever sit in on user testing sessions?

Karen: Actually pretty regularly, and likewise, the researchers are always eager to use data in their research and combine more of the quantitative piece with some of the qualitative work they’re doing. It happens both ways. And sitting in on some of those sessions is really eye opening. If there’s analysts or data scientists out there who have never done that, it’s a really great experience for everyone. We do that all the time.

Data: it’s all about people and behaviors

Geoffrey: You got a great reaction to your recent talk at Inspirefest, and to summarize it very crudely, your thesis is that data is fundamentally about people and behaviors. Could you just give us some insight into that?

Karen: The general theme was artificial intelligence and real world data. Reflecting back on my career, I thought it would be interesting to pull out concrete examples of analysis and data explorations that I conducted to craft a message, which is that data is fundamentally about people and their behaviors. I started all the way back in PhD and I described an analysis I did with a European mobile operator where it was basically 300 million URLs and six million search queries all generated by millions of European mobile users all the way back in 2006. This is pre-iPhone. I was able to look at the topics they searched for in search engines like Google, and we found there was an awful lot of adult-related content back then.

Then when I moved to Telefonica, I worked with colleagues there understand more about smartphone notifications. All of these apps like WhatsApp and Facebook, they proactively push these notifications towards us to make sure that we don’t miss out on key messages and events. We were really intrigued with how many of them people actually have to deal with. And more interestingly, how they handle them and what’s effect notifications on them.

So we built this logging app that we installed on their phone. It sounds a bit creepy, but we did get consent. And it tracked all of the notifications they received and the apps that generated those notifications. We were also able to tell things like whether the ringer was on silent mode or vibrate mode. It enabled us to not only look at how many notifications people received, but also how it took 3.5 minutes on the weekends for people to react to notifications from WhatsApp, which is super fast. Having the phone in silent mode had no effect on response times. People have their phone in silent mode a lot and yet they check their phone every two minutes just in case they receive a message. I have my phone permanently in silent mode, and I’m constantly checking it.

The last story was actually recent work from Intercom. We did a really cool analysis looking at emoji use in business messaging. We had a data set of one million conversations over the summer of 2015 and a million over the summer of 2016, and we looked at emoji trends.

We found that emoji usage has increased four times over in that one year. We looked at top emojis, but more interestingly, we looked at mirroring. Do businesses mirror the type of emojis that their customers use? The answer is no. They probably should, but end users, people like you and I, were using highly personal emotive emojis. It was all faces, hearts and gestures. There was no objects or business-based emojis. It was some really nice insight about how people try and augment our online experiences with all these new features like emojis and gifs, to mimic some of what we have in our real, physical lives.

It was nice to go back and reflect on some of the earlier pieces of work that I did. And to pull out that horrible looking table and then try and build a beautiful visualization around it.

Geoffrey: Lots and lots of reflection.Karen, thank you so much for joining us. It’s been a blast.

Karen: This was fun. We’ll have to do it again sometime.