Unlocking customer value with data: How we developed the Customer Milestones Framework

For any sophisticated software product, it can be surprisingly hard to figure out exactly how your customers actually adopt and use your product. At a certain scale, customers will find a seemingly infinite number of workflows, implementations, and use cases. Which is great – if people are getting value out of the tool, then the software is doing its job. Right?

Well, to ensure the software continues to do its job as your users’ needs change and evolve, you need to have a very deep understanding of how those people are getting the value they’re paying for. But those joined-up, holistic insights can be frustratingly elusive.

At Intercom, we conducted a huge, company-wide research project focused on understanding and optimizing our customer journey to ensure we had the right insights to deliver better experiences and long-term success. This involved reflecting on our existing customer journey and identifying an ideal journey where our customers are getting set up successfully, deriving clear value from the product, and expanding through new use cases.

We wanted to clearly identify those usage milestones that marked out the most successful users, so we could help all our customers to find the same value. We called it the “Customer Milestones Framework” – a data-driven, actionable plan to drive deeper usage and engagement. It also helped all our teams keep aligned to ensure a coherent, customer-centric approach to building, marketing, and selling our product.

How did we go about it? Read on to find out.

Introducing the Customer Milestones Framework

At the heart of the framework are four distinct customer milestones:

• Setup: The initial setup stage to go live with Intercom currently consists of seven primary steps, ranging from installing the Messenger to configuring automation features.
• Activation: This milestone involves the early stages of customer interaction and indicates that the customer has started to get initial value from the product. We define it as the customer reaching a certain threshold of conversations within the first three months.
• Utilization: This milestone represents meaningful product engagement, wherein customers are experiencing the core value of the product.
• Maturity: This stage indicates that the customer is using the platform’s full capabilities to get maximum value from the product. We look at five categories spanning across our product: support channels, automation, support ops, interoperability, and reporting.

Customers who successfully complete these stages are more likely to remain Intercom customers and expand over time.

Identifying the milestones

A project of this size and complexity required a multidisciplinary team to get the full range of insights required. Three data scientists and one product researcher partnered with one of the product leaders in the Growth/Sales team.

“Working backwards, we built up a coherent pathway from setup to retention”

But where to start? We worked backward from customer retention and deep product usage to define the milestones. We defined thresholds of meaningful usage beyond which customers are more likely to retain and expand. Each product area had its own success metric to enable R&D teams to identify customers continually achieving value from Intercom.

Working backwards, we built up a coherent pathway from setup to retention – identifying the setup steps that were correlated with improved activation rates, the activation threshold that led to improved utilization and ongoing engagement, and finally the hallmarks of utilization that led to mature and extensive usage of the platform.

Maturity milestone

What does mature usage look like in practice? What features are used by the most successful customers?

To develop the maturity model, we combined qualitative and quantitative methods. We first asked Product Managers to map out what “mature” product usage meant to them. Through regular research and talks with customers, we’ve built up a strong opinion on what features mature or sophisticated customers use, versus what features are used by less mature users.

“It was crucial to strike the right balance and arrive at a milestone that represented our customer base”

For instance, we hypothesized that customers using apps and integrations are more mature than those who are not. Then we wanted to test these assumptions quantitatively. The identified features were validated using a random forest model, which provides a feature list of actual product features ordered by importance scores. The list consisted of over 50 features ranging from certain Inbox or Messenger configurations to add-ons or feature usage. They were ordered by the degree of how good of a predictor of retention the feature was.

However, the importance scores were somewhat skewed towards our niche features used by a narrow group of Premium customers. To address this issue, we validated the model’s output against an adoption benchmark (how prevalent the usage of a feature was). Some features were so broadly adopted that they didn’t offer any discernment between customers (e.g. customising look and feel of the Messenger homescreen), while others focused on a narrow group of customers that in the eyes of the model made them look unique (e.g. using one of our Premium features, e.g. custom SLA rules). It was crucial to strike the right balance and arrive at a milestone that represented our customer base. We landed at five categories spanning our entire product:

• Support channels
• Automation
• Support operations
• Interoperability
• Reporting

We distinguished two levels of usage: basic or mature. Today we measure the percentage of customers who reach maturity in three or more categories.

Utilization milestone

For the utilization milestone, our goal was to define a threshold of meaningful product usage relative to the customer’s size, as some of our healthy and well-retaining customers might not meet deep usage thresholds if they were small companies.

We decided on a metric that was relative to what customers purchased to better reflect the level of their engagement with our product. We set a threshold of support conversations that customers resolve in a month per Inbox seat they purchase. By adding the relative component of seat volume, we ensured that this metric scales well with the customer’s size.

Activation milestone

We had previously defined an activation metric (support conversations threshold in the first three months of tenure) that was broadly adopted in the company.

“We validated that activated customers were less likely to become dormant”

We felt it was an important milestone on the customer journey and we wanted to ensure that it fit in with the framework we were developing. Therefore, we validated that activated customers were less likely to become dormant and more likely to continue using the product in a meaningful way.

Setup milestone

To define the setup milestone, we also combined qualitative and quantitative methods. We collated from our R&D and Sales partners what a successful setup looked like and verified each of them by measuring correlation with activation rates.

Similar to the maturity milestone, we had to balance niche setup actions with the routine ones that weren’t unique enough and didn’t correlate with improved activation.

Operationalizing the framework

The overarching goal of the project was to create a company-wide framework that would be used to drive customers towards deeper product engagement.

To bring the Customer Milestones Framework to life, we created a cross-functional group across R&D, Sales, Marketing, and Customer Support. The team developed strategies to drive progress across the customer journey and maximize customer value at every stage. Some key initiatives this cross-functional group tackled included:

• Team Activation redesigned and streamlined the onboarding experience for our self-serve customers to help drive better setup of Intercom.
• Our Customer Support org tailored their engagement with customers to help them maximize their setup by recommending a “next best step” action.
• The Customer Success team adopted a more data-driven approach to increasing product usage and adoption across all four customer milestones.
• Marketing built new content and campaigns focused on driving customers to activation and maturity.

The Customer Milestones Framework has allowed us to make more informed decisions on R&D investment, as well as design support and sales interventions to help our customers realize the true value of Intercom. By working collaboratively with various teams across the organization, we have been able to make a positive impact on product engagement and create a single, unified language and system around the customer lifecycle.

Lessons learned and key takeaways

Throughout the course of the project, we derived valuable insights about our customers and what it takes to coordinate a company-wide project. If we were to do it all again, there are two things we would have done differently:

• Establishing a framework review process: How do you know if the framework, well, works? We didn’t set out in advance how we would approach a review of the framework, and we realize now that planning for that review from the beginning would have provided clearer guidelines and instilled confidence in the model’s effectiveness and longevity.
• Data operationalization: This was a big one – obviously it was hard to predict in advance how long the rollout of the Customer Milestones Framework would take, or what sort of issues would arise as we implemented it for different teams. It was longer and trickier than we anticipated – we had to collaborate with other teams (data infrastructure and business system teams) to ensure that accurate data was available in a timely manner, and we also had to dedicate time to train our colleagues on utilizing this data effectively in their daily workflows. What should have felt like the end of the project ended up taking up a big chunk of the time devoted to it, but it helped us build a strong foundation for the project’s success.

Developing the Customer Milestones Framework has given us a powerful platform for ensuring ongoing customer value and helped with customer retention.

The greatest advantage of the framework, however, is the way it has given us a unified structure and language that has been adopted across the entire company – it has kept us all on the same page in our shared goal of deepening product engagement and delivering a valuable customer experience. We’re excited to double down on this in the coming months to drive even more value to our customers and our business.

The Customer Milestones Framework was a huge cross-functional effort, led by Intercom’s Research and Data Science team. I’d like to acknowledge in particular my key collaborators on this project – Louis Ryan, Lynsey Duncan, and Eamonn O’Neill – for their stellar work devising and delivering the framework.