Navigating AI metrics

How do you lead a support team in this new world with AI metrics?

The technology is amazing, but our assumptions and processes for understanding and leveraging AI metrics are very different from traditional support metrics. Our new CX Score is the perfect example.

This post originally featured in our AI-first customer service newsletter, The Ticket. 👉 Subscribe and join the 28K+ customer service professionals who receive industry news, tips, and career guidance directly on their LinkedIn feed.

Two months ago, we launched CX Score – a new way to analyze every single conversation and give you a complete view of your support experience. (Check out our VP of Product Brian’s post on our /ideas blog for a deeper dive.)

It’s a powerful tool, and I was really excited as someone who’s battled with CSAT survey mechanics, teammate exclusion processes for CSAT, and the nagging truth that this is only a small portion of our volume. We’ve learned a lot navigating CX Score – lessons that will apply to all AI metrics – and here are two key takeaways:

Lots of data calls for new processes

One of the first things we noticed was the sheer amount of data.

For better or worse, CSAT was a small enough sample size to review every comment – particularly unhappy ones. Our QA team would read and categorize each response, and follow up with customers. Managers would read most comments for their team (~15 in total per manager), and discuss in 1:1s.

But what do you do with 1,600+ reviews across the org? This is the reality of AI metrics, and when you have more data than ever before, the old processes don’t scale.

• We briefly tried reviewing all unhappy CX ratings.
• We tried taking a sample, but this felt just as limited as CSAT.
• We exported the trends and conversation data back into an LLM for analysis, but without in-depth prompting the results were only okay.

So, what have we found works?

Because CX Score is great for reviewing trends, we use it to measure week-over-week performance for both Fin and as a team wide KPI for human support. We also use CX Score to review specific targeted areas, like a new hire’s conversations on a certain product area. And to review the customer experience for a group of customers, or to analyze a customer’s entire case history so we can lean in at the customer level.

The complexities of AI mean we won’t always know the “why” – and that’s ok!

People naturally want to know “the why” – especially support folks. When we started using CX Score, one of the biggest challenges was the team wanting to dig deeper into why a specific score was given.

While the score provides a great overview, people wanted a detailed, step-by-step explanation. But LLMs are mostly a “black box” – especially to the everyday person. As AI becomes more and more ingrained in our work, we’ll need to accept not always knowing every detail.

This required a mindset shift for both the wider team and leadership as we moved into a world of AI metrics:

• Focus on the outcome vs. the process: We celebrated the positives and highlighted the insights and actions previously impossible with only CSAT.
• Don’t compare to humans: We challenged and reminded the team that many of the unknowns of AI are equally true with humans. Even with a large survey, we never know for sure how customers feel.
• Acknowledge emotions: People need time and space to process feelings. Our Ops Manager William would poll the leadership team in our weekly ops meeting, asking, “In one word, how did the CX Score make you feel last week?” This gave managers the space to share wins and challenges.

What’s next

We have a ways to go to revamp how we work, and adjust our collective mindset for AI tools and metrics. But the pros highly outweigh the cons, so I encourage you to jump in and start experimenting.

Lastly, this technology is improving very quickly. Just yesterday we added deeper AI explanations and additional attributes to explain the CX Score and aggregate summaries across topics. I’m excited to try it out!

📩 Subscribe to The Ticket here – a bi-weekly LinkedIn newsletter delivering key insights for customer service professionals in this time of mind-blowing change.