The AI agent buyer’s guide: How to make the right choice for your support needs

Working with AI is no longer an option for customer service teams – it’s a necessity. With an influx of new entrants promising powerful AI agents, choosing the right solution has become increasingly complex.

Making the right choice is a challenge for today’s support leaders for a number of reasons. Solutions range from basic chatbots to much more sophisticated AI agents, but elaborate marketing claims make it hard to tell them apart without hands-on testing.

“To make an empowered decision, support leaders need a systematic way to cut through the noise”

Integration requirements vary widely, cost structures are complex, and it’s difficult to predict which solutions will remain viable long-term. Faced with a market that’s moving incredibly fast, you need to trust not only that your vendor can keep pace with what’s current, but also that their solution truly aligns with your unique business needs, company values, and customer expectations.

To make an empowered decision, support leaders need a systematic way to cut through the noise. In this comprehensive guide, we detail exactly how to evaluate AI agents based on what matters most. To distinguish between truly powerful performers and basic tools wrapped in shiny (thin) wrappers, we’ll walk you through how to:

• Audit your support profile to help you understand exactly what you need from an AI agent.
• Identify your evaluation criteria to ensure the options you’re considering tick all the boxes.
• Test different AI agent options by applying a rigorous framework that’ll reveal how they really stand up.

Choosing the right AI agent to keep ahead of your competitors and deliver excellent customer service is crucial. Let’s explore everything you need to know to make the best decision.

Understanding AI agents

Automation in customer service has evolved dramatically with the advent of generative AI and large language models (LLMs). While the conversational interface looks the same, AI agents fundamentally differ from the chatbots that came before them, and understanding how they operate is important as you try to assess the options in market.

Key differences between chatbots and AI agents

 

• Traditional chatbots rely on rigid, rules-based systems, using decision trees and pre-scripted responses to simulate conversations. They require extensive manual configuration to detect keywords and deliver relevant manually curated responses.
• Modern AI agents are powered by LLMs and can understand natural language, interpret and remember context, and generate human-like responses. They can process and synthesize vast amounts of information from sources such as knowledge bases, enabling them to automatically understand and resolve many informational customer queries in natural conversations.

This technological leap has brought with it massive efficiency gains. For instance, Intercom’s Fin AI Agent resolves an average of 51% of customer queries out of the box with 99.9% accuracy.

As AI agents become more sophisticated, they are expanding beyond conversations and will be able to take actions and handle even more complex tasks, further transforming customer service.

Audit your support profile

Before you dive into exploring options from different vendors, take some time to understand what your “support profile” looks like. Performing an analysis of the queries your team encounters every day will help you identify exactly what your needs are and where an AI agent can help.

Consider the following four factors:

Support query volume

The number of queries you receive will be a major factor in your choice of AI agent. You’ll need to compare the AI agent’s potential resolution rates, pricing models, and their ability to work with your human support team.

Support channels

Where do your customer queries come from? If you’re working across more than one channel, as most support teams are, you’ll need to know the AI agent’s capabilities for handling your queries and ensuring a consistent customer experience.

Query complexity

Informational queries are at the heart of customer service – for most businesses, the majority of tickets involve customers seeking answers to questions. That’s where AI agents already excel.

Take a close look at how your team currently handles queries about things like product features, pricing, policies, and technical specifications. Do seemingly simple questions often require more nuanced responses? Are your team members having to combine information from multiple sources to piece together answers?

“As AI agents evolve, they will become more and more capable of handling personalized and action-based queries”

Use these insights as you start evaluating different AI agents. You’ll discover when you reach the testing phase that different options vary significantly in their ability to resolve informational queries – there can be significant variation in accuracy and resolution rates.

As AI agents evolve, they will become more and more capable of handling personalized and action-based queries, so you should also consider whether the vendors are discussing features such as this in their product roadmaps.

Knowledge content health

The current state of your support content will impact how quickly and effectively you can deploy an AI agent. Consider these questions when assessing your knowledge base:

• Is the content up-to-date?
• What major gaps need to be filled?
• Is everything centralized in one location, or is it scattered across different platforms?
• Do you follow a clear process for maintaining your content and keeping the information current?
• How easy is it to find the information you’re looking for?
• Are there valuable knowledge sources outside your knowledge base that could enhance your AI agent’s capabilities, such as website content, internal process documents, or PDFs that explain product features?

Many teams find that preparing to implement an AI agent provides them with a great opportunity to audit and optimize their knowledge content. A consolidated, well-maintained knowledge base not only makes it easier to train your AI agent, but also enables it to provide the most accurate, consistent responses to customers as soon as it’s up and running.

It’s also worth noting that modern AI agents’ ability to draw from multiple sources, like pricing pages, technical documentation, and product guides, enables them to construct richer, more comprehensive answers than traditional support tools. This empowers support teams to leverage content from across the entire organization, expanding their influence and ability to deliver expert assistance on any topic the company has documented.

Identify your AI agent evaluation criteria

Once you’ve understood what your team’s unique needs are, you can start assessing your options. This evaluation process involves two distinct parts:

1. Your internal adoption requirements.
2. The customer-facing capabilities you need.

Internal adoption considerations will help you determine how smoothly the AI agent can be implemented and maintained by your team, while customer-facing features will ensure the solution can deliver the best level of service for your customers.

1: Internal adoption requirements

There are four foundational areas to consider, namely:

• How long the AI agent will take to set up.
• How easily it can integrate with your existing support platform.
• How much it costs.
• How it handles data privacy and security.

Setup time: Choose an option that’s quick to get started with

Working in customer service is busy (to put it lightly), and most teams don’t have a spare minute, let alone hours, to dedicate to a complex AI agent setup process. Setup times can range from a few minutes (in the case of Fin AI Agent) to days, or even weeks – so be sure you know what you’re letting yourself in for.

“The sooner your AI agent can assimilate with your team’s processes, the sooner you can start to see real results and return on your investment”

Consider the resources you have on hand to get your AI agent up and running: which team members can you spare, and for how long? How much time can you give to team training? If the answer is “not much”, it’s worth prioritizing an AI agent that can hit the ground running.

The sooner your AI agent can assimilate with your team’s processes, the sooner you can start to see real results and return on your investment.

Find out:

• How easy is it to get started with this AI agent?
• Will this AI agent require training before it can begin to bring real value to your team?
• What kind of resolution rate can you expect to see right out of the box – and how will it improve over time?
• Will the AI agent require ongoing technical configuration to operate effectively? If so, does your team have the necessary resources?

Ease of integration: Get up and running without having to overhaul your current support platform

Every support team has carefully planned processes and workflows that are optimized for efficiency and an excellent customer experience. It’s natural to fear that bringing in a new AI agent could disrupt the way the team operates.

An AI agent should work with – not parallel to – your current setup. Some AI agents operate as add-ons to whichever platform you’re using, while others are native to their platforms.

Find out:

• How will this AI agent work with your existing tech stack?
• Do you need to purchase other tools alongside the AI agent to maximize effectiveness?
• What kinds of integration capabilities will you need to get the most from your AI agent?

Cost: Pick a pricing model that works best for you

When it comes to AI agents, it’s incredibly important to understand the kind of pricing model that will provide the best ROI for your team.

AI agents are being priced in many different ways, but these models essentially boil down to two pricing philosophies:

• Pricing against outcomes: This approach is based on measurable results that impact your team performance. An example would be pricing against resolutions, which is what we’ve chosen to do at Intercom. This means you only pay when your customer has received a satisfactory answer to their question without having to be passed to your support team. This approach is value-based, measurable, and makes it easy to tell the kind of ROI your AI agent is providing to your team.
• Pricing against usage: This approach involves pricing based on API requests or messages. It can often be difficult to ascertain ROI when your AI agent is priced against these metrics because they don’t account for actual resolutions. It’s hard to be sure whether your customer got the answer they needed, or just left the conversation frustrated.

“Our most successful Fin customers think about the return on investment through two lenses: increased bandwidth and cost efficiency”

Consider what will ensure the highest ROI for your team by comparing prices against the cost of adding headcount, and against the savings you’ll gain from improving team efficiency and dedicating more time to driving customer value and satisfaction.

At Intercom, our most successful Fin customers think about the return on investment through two lenses: increased bandwidth and cost efficiency. Their goal is to run an efficient organization where every dollar is put to good use. Here are two important things to consider when quantifying the ROI of AI first customer service:

1. Purchase price vs long-term value: When you’re assessing AI agents, it can be tempting to look solely at the price per resolution. But what moves the bottom line the most – and delivers the greatest ROI – is actually the tool with the best resolution rate and performance rate.
2. Total cost of ownership: To get the full picture, you also need to think about the total cost of ownership, which includes all of the “hidden” costs that come with adopting any new tool.

Download a copy of our new guide, The New Economics of Customer Service, for more insight into how AI has shaken up the linear growth model of customer service and revolutionized how teams think about the ROI of AI.

Find out:

• What metric are you using to determine how much your team will pay for the AI agent?
• How are you defining this metric?
• What is your cost to serve? How does it compare to the price per unit, i.e. per resolution, per message, or per deflection, depending on your chosen AI agent?

Protecting your customers: Be upfront about your data privacy and security requirements

Data privacy and security is a priority for every company, but depending on the nature of your business, it may be your number one concern.

AI agents use LLMs to respond to the questions they receive, meaning the information your customers offer will be processed through any of the increasing number of LLMs available, whether it’s Claude 3.5, GPT-4o, Gemini, or any other.

In order to assure your customers that their data will be safe, it’s important to understand the LLM your chosen AI agent works with, and how it handles your data. Below are some questions you can ask to dive a little deeper into the suitability of an AI agent for your business.

Find out:

• How will your customer data be used? Will it be shared with, or stored by, the LLM provider?
• Is your customer data being used to build AI models by your current provider? Are you comfortable with this?
• Will customer messages be encrypted?
• Does the AI agent vendor have a partnership with an LLM provider?
• Where is your data hosted, and will the location affect your ability to use the AI agent?
• What legal agreements, including NDAs, are required before testing the AI agent with real customer data?

Pro tip: Start discussions about NDAs and data protection agreements early in your evaluation process. Many companies find that getting these agreements in place takes longer than expected, which can delay testing and implementation. Having these conversations upfront will help with a smooth transition from evaluation to deployment once you’ve decided which AI agent you want to go with.

2: Customer-facing capabilities

After ruling out any AI agents that don’t suit your internal requirements, you can move onto evaluating the quality of support the AI agents can actually deliver. There are a million ways vendors are marketing these features, but ultimately, characteristics of exceptional frontline support fall into these categories:

• Knowledge
• Behavior
• Actions
• Insights

Consider what each AI agent on your list offers within these categories and how you can leverage this to provide the best experience for your customers.

Knowledge: Intuitively deliver the most accurate, comprehensive answers

All AI agents rely on quality knowledge content to function. But how they use this content to deliver results can differ drastically.

Strong performers operate like your most experienced team member, seamlessly drawing from your entire knowledge ecosystem to craft complete, accurate answers. They can instantly learn from all available sources, including help centers, internal documents, PDFs, and URLs and process this information at lightning speed. Most importantly, they’re able to come up with responses to even the most complex questions by combining relevant information from multiple sources, ensuring customers receive comprehensive answers that solve their problems completely the first time.

Fin has the ability to combine knowledge from multiple content sources to create tailored answers for your customers.

Not all AI agents can handle this level of complexity. These options tend to offer single-source responses rather than true information synthesis. They typically require lengthy training periods before delivering real value, need constant manual updates to stay current, and frequently miss crucial context by failing to connect related information across different sources. This limitation will lead to incomplete answers, frustrated customers, and increased workload for human agents who need to fill in the gaps.

Find out:

• What types of knowledge sources can the AI agent integrate with (help centers, internal documents, PDFs, URLs)?
• How quickly can the AI agent process and begin utilizing new information?
• Does the AI create answers by combining information from multiple sources, or is it limited to single-source responses?
• How easy is it to update or modify the AI’s knowledge base as your product evolves and processes scale?
• What mechanisms are in place to ensure the AI provides accurate, up-to-date information?

Behavior: Mirror your human team to ensure a seamless interaction experience for customers

The spectrum of AI behavior control ranges from basic tone-of-voice settings to comprehensive workflow management. Some AI agents operate with fixed responses and limited flexibility, while others can be trained like human team members – learning your policies, adapting their communication style, and making nuanced decisions about when to handle issues versus when to escalate them. The key difference lies in how naturally these behaviors can be implemented and modified.

“When an AI agent can be trained like a human team member, it doesn’t just handle inquiries; it becomes an integral part of your customer experience strategy”

Consider your current support team guidelines: What policies must be strictly followed? Which situations require human intervention? How do you handle multilingual support? Your AI agent’s ability to understand context, follow workflows, and seamlessly integrate with your existing processes will determine its effectiveness as a true front-line support solution.

When an AI agent can be trained like a human team member – understanding policies, detecting customer sentiment, and adapting its behavior accordingly – it doesn’t just handle inquiries; it becomes an integral part of your customer experience strategy. The right combination of guardrails and flexibility means you can trust your AI agent to represent your brand while maintaining compliance and consistency.

Find out:

• How can you control the AI agent’s tone and communication style to match your brand voice?
• What mechanisms exist for creating and modifying behavioral workflows and rules?
• How reliable and robust are the mechanisms to ensure workflow and policy adherence and prevent unauthorized or mistaken actions?
• How does the AI agent handle multilingual support and cultural nuances?
• Can the AI agent detect and appropriately respond to customer sentiment?
• How does the AI agent determine when to resolve issues versus when to escalate to human support?
• Can you establish different behavioral rules based on customer segments or channels?

Fin supports multilingual interactions, automatically translating your content in real-time to match the customer’s language.

Actions: Take meaningful steps independently to resolve customer issues

AI agents are developing fast, and the next phase will see them increasingly able to take independent actions in response to customer queries.

This capability is still emerging, but forward-thinking support teams are already planning for a future where AI agents will progress from information retrieval to executing complex, multi-step processes.

When considering AI solutions today, it’s crucial to understand vendors’ plans for developing these capabilities. You want to be sure their roadmaps are aligned with your future needs so that the AI agent you choose now continues to serve you in the long-term.

Find out:

• What is the vendor’s roadmap for developing action-taking capabilities?
• Which third-party integrations are they prioritizing?
• How are they approaching security and authorization for automated actions?
• What is their timeline for implementing these features?
• How do they plan to give customers control over AI permissions?
• What safeguards will they put in place for automated actions?

Fin will be able to retrieve customer data and provide answers specific to each customer, like checking recent orders.

Insights: Get a deeper understanding of performance metrics to drive better results

The ability to measure and improve your AI agent’s performance isn’t just about tracking metrics – it’s about understanding the true impact on your customer experience and business outcomes. For support teams, comprehensive insights mean the difference between flying blind and having a clear view of service quality across every single customer interaction.

When choosing your AI agent, consider what kind of reporting you’ll need for the following areas:

• Determining the ROI of your AI agent: You’ll need a reporting system that offers robust insights into how the AI agent is affecting your most important metrics.
• Areas for improvement: Some AI agents have the capacity to improve as they ingest more of your support material, and as your team strengthens your support center and and optimizes workflows to boost its performance. A solid reporting system will indicate the areas where the AI agent could be offering more value, allowing you to maximize its strengths.
• Customer satisfaction (CSAT) measurement: Look for systems that are moving towards providing comprehensive CSAT measurements across both human and AI interactions. Soon, the most advanced solutions will be able to offer AI-generated CSAT scoring that analyzes 100% of conversations, giving you way more insight than you currently get from a few filled-out customer surveys.
• Unified performance view: Since humans and AI will work together to handle support, you’ll need holistic reporting that shows how both are performing. A unified view will help you understand the overall health of your customer service operation and how different parts of your support system complement each other.
• Sharing results with the wider company: As every customer service manager knows, pulling these insights is only half the work – the rest is sharing the reports with management and the wider company so everyone can get behind the support team’s efforts and appreciate the value AI is bringing to the customer experience.

The analytics capabilities of AI agents can vary – from basic resolution rate tracking to sophisticated AI-powered analysis of conversations. While some solutions offer only standard metrics, advanced solutions are progressing to a point where they’ll soon be able to evaluate customer sentiment, measure true resolution rates, and provide detailed quality assessments across both AI and human interactions.

Find out:

• What kind of reports are available?
• What metrics are available for comparing AI and human agent performance?
• How will the reporting system work with your current reports?
• How easily can reports be shared with stakeholders across the organization?
• How granular are the quality analysis capabilities?
• Can the system identify specific areas needing content or workflow improvements?

Test different AI agent options

It’s relatively easy to make an AI agent look impressive in a controlled demo, which we call the “AI demo problem.” Similar to self-driving cars that blew people away with their performance on closed circuits, many AI agents can look deceptively fast and impressive in a demo environment answering basic questions.

“Before you make the final call on which AI agent is the best fit, make sure you’re confident about how it actually performs”

To truly test its capabilities, you need to see how it handles real-world challenges and push it with complex or ambiguous questions – and know what to look for in its responses. A good AI agent will respond accurately or admit it doesn’t know. A poor AI agent will confidently provide misinformation.

Before you make the final call on which AI agent is the best fit for you, make sure you’re confident about how it actually performs. You want to be sure that the tool you’re bringing on board will truly be a great addition in reality and not just “passable.”

Here are some steps to thoroughly assess how an AI agent will perform in practice.

Pro tip: Remember that an AI agent’s performance is only as good as the content that it’s trained on. Before running any tests, review the help articles, documentation, and data sources your AI agent will be drawing from. Are they up-to-date? Do they cover all the necessary information? Are there any gaps in your content that need to be filled? Having comprehensive, well-organized content is essential for accurate responses, and will impact the results you get.

Step 1: Prepare your test cases

To truly test how an AI agent will perform in your environment, you need to get specific.

To ensure you’re covering both common and current scenarios, gather 10-20 of your most frequently asked questions and a selection of your most recent queries. This combination should hopefully give you a range of questions that vary from basic to complex, and focus on different areas of your product or service offering.

To go more in depth, you might also want to prepare:

• Complex queries that typically require multiple touchpoints from different team members.
• Phatic or vague queries that don’t contain any “real” information and require further clarification from the customer to resolve.
• Queries spread across multiple turns.
• Edge cases that have been difficult for your human team to resolve.
• A few sensitive scenarios, such as billing disputes and cases where customers have become angry.
• Examples of queries in different languages, if you provide multilingual support.

Step 2: Test variations of the same question

Once you’ve gathered your list of questions, prepare a few variations. Real customers rarely ask questions in the exact same way, so you need to ensure your AI agent can handle different types of communication.

For each test case you prepared in the first step, try:

• Difficult questions that require information from multiple sources to answer.
• Different phrasings of the same question.
• Incomplete or fragmented queries.
• Questions with typos or grammatical errors.
• Various levels of formality.

Step 3: Simulate real conversation flows

Customer conversations rarely follow a straight line from question to resolution. Put your AI agent through realistic scenarios by:

• Starting with vague queries that require clarification.
• Testing follow-up questions to see if context is maintained.
• Introducing new information mid-conversation.
• Changing the topic or circling back to previous points.
• Expressing frustration or confusion during the interaction.

Step 4: Challenge the guardrails

Understanding your AI agent’s limitations is just as important as knowing its capabilities. You want to be sure the AI agent won’t answer something it shouldn’t. Test its boundaries by:

• Requesting information it shouldn’t have access to.
• Asking about competitors or sensitive topics.
• Trying to bypass security measures.
• Using slang, technical jargon, or industry-specific terms.
• Asking it to perform an action it hasn’t been configured for.

Step 5: Document and analyze

For each test scenario, make note of:

• The exact query or action attempted.
• The AI agent’s response or action taken.
• Response time and number of steps required.
• Any unexpected behavior or errors.
• Areas where human intervention was needed.
• Opportunities for improvement.

As you’re working through the testing process, remember that the goal isn’t to find an AI agent that passes every test perfectly, but rather to understand exactly how it will perform in real-world situations with your customers. Pay special attention to how it handles uncertainty, maintains conversation context, and knows when to escalate to your human team.

Speaking of your human team, while it’ll require some planning and dedicated time, it’s a good idea to get them involved in this testing process. If possible, have multiple team members run through these tests independently. Different people will interact with the AI agent in different ways, which will help you uncover potential issues from various perspectives.

Getting your human agents involved is also a great way to bring them along with you on the journey and get their input on which AI agent is the best fit. As the experts who are on frontlines, they’ll be the ones working most closely with it every day, so it’s critical to secure their buy-in.

Welcome aboard, AI agent

Hopefully, this guide has set you up with everything you need to make a confident decision about which AI agent is best for you.

Taking the time to thoroughly evaluate your options will help you lay the foundation for a transformative shift in how you serve your customers. When implemented thoughtfully, an AI agent can revolutionize your customer experience by providing instant, accurate resolutions, while simultaneously empowering your human team to focus on complex, high-impact work.

The result is a win-win-win: happier customers who get faster resolutions, more engaged support teams doing meaningful work, and a business that can scale efficiently while providing world-class service that leaves competitors in the dust.

How transparency can help your customers get more out of AI support

We’ve spoken a lot about preparing your support team for a new way of doing customer service with AI, but what about preparing your customers for a new way of receiving customer support?

According to a recent Gartner survey, 64% of customers would prefer it if a business didn’t use AI for customer service. The reason behind this is something we can all relate to – interacting with chatbots has historically been a shocking experience.

It’s not necessarily that the technology was bad, but by and large, chatbots weren’t set up well, which made interacting with them clunky and frustrating.

Luckily, we’ve come a long way – even since that Gartner research was conducted – and next-gen AI agents like Fin are starting to have a positive impact on customers’ perception of automated support.

AI is only as good as it’s set up to be, however, so support teams have the responsibility of really making the experience great for customers. Here are a few quick tips to help set your customers up for success when interacting with your AI agent.

 This post originally featured in our customer service newsletter, The Ticket.

 

👉 Subscribe and join the 25K+ customer service professionals who receive industry news, tips and career guidance every two weeks in their inbox.

Tell your customers how to talk to your AI agent

Perhaps due to poor experiences with previous generation chatbots, many people don’t actually know how to interact with AI agents. Instead of asking a question in a normal conversational manner, they resort to single keywords or basic phrases, like “need help.”

“Chatting in a natural, conversational way works best”

AI agents are pretty smart, but they’re not mind readers. Not having enough information is going to cause the bot to draw a blank and frustrate the customer, so it’s important that customers know how to communicate to get the best results.

Chatting to Fin in a natural, conversational way works best, so we set this prompt at the beginning of a customer’s chat with Fin that explains this in a really brief way:

Hi [First Name], you’re speaking with Fin AI Agent. 👋

 

I can do much more than chatbots you’ve seen before.

 

Tell me as much as you can about your question, and I’ll do my best
to help you in an instant.

We worked closely with our conversation designer to get this right by running a couple of A/B tests and tweaking the copy until we landed on something that clicked for our customers.

This simple prompt has helped customers have much richer interactions with Fin and a better overall experience, so I’d highly recommend trying something like this if your customers are struggling to get what they need from your bot.

Lead with your AI agent, but make it easy to access human help

Customers are getting used to AI agents being the first to respond to their queries, but many still want to know they can easily access a human’s help. This is why it’s important to make the handover from AI to human as seamless as possible when a customer requests it.

“The key is limiting the effort the customer has to put in”

When one of our customers opens the Messenger and asks a question, we’ve set Fin to answer first as much as possible in our automated flows. If Fin can’t answer, the customer can select “I need to speak to a human.” A few clarifying questions then help the human agent understand the customer’s issue.

The key here is limiting the effort the customer has to put in. We conducted some research and found that after just four clicks, a customer will drop off if they’re not connected to a human.

We used to talk about “deflection” in support, but abandonment rate – which is what we’re referring to here – is not good. Customers just going away rather than receiving help is not something you want as a business.

As customers continue to have good interactions with AI agents and develop more trust in them, we’ll see more and more queries resolved without needing input from the team.

Set expectations upfront

There is some debate around this, but I firmly believe support teams should let customers know when they’re speaking with an AI agent.

“Always be honest with customers to foster and maintain trust”

Some teams like to “humanize” their AI agents with a human photo and name, but in my opinion, we’ll only really help our customers get more comfortable with this technology if we’re transparent about what it is.

We’re all working towards making automated support better and a big part of this is bringing our customers along with us on this AI journey. My advice is to always be honest with customers to foster and maintain trust.

Response Time: Vol. 39

You satisfy your customers, but can you satisfy our curiosity?

With Sam Barrett, Head of Customer Experience at Runna.

Please tell us a little bit about your company and what you do there.
I’m Sam, Head of Customer Experience at Runna. We’re a running training platform that aims to make running easy, enjoyable, and effective for all.

What word or phrase in customer service jargon should be retired?
“Circle back.”

Which celebrity would be really great at your job?
Mikel Arteta.

What’s the most valuable thing that working in customer service has taught you?
Be kind to EVERYONE! It doesn’t matter their role, title, or experience. Be kind and lead with care.

Describe the essence of great customer service using only three words.
Tell your friends.

Which movie robot would you choose as your AI sidekick, and why?
WALL-E. He’s kind, friendly, and supportive and whilst he doesn’t steal the show, he deserves a lot of credit for getting the job done. Everyone needs a WALL-E!

What can you do that a bot will never be able to replicate?
Provide hyper-personalized and individually specific coaching guidance to first-time runners.

How do you go the extra mile for your customers?
I frequently share my personal running routes with our customers. I’ve spent hours finding the best routes where I live, and when I see a customer based in my hometown I go above and beyond to share all the hidden gems.

Do you identify more with the title “customer support,” “customer service,” “customer success,” or “customer experience,” and why?
We identify most with “customer experience.” We wanted to encapsulate the end-to-end experience of our customers and show CX as a value-add, rather than a cost to the business. “Customer service” sounded too transactional for us and I’m glad we made the change.

What’s the one piece of advice you would give to your peers in the customer service industry?
Bring your team along with you on the journey! Especially when you are making big decisions and changes. It’s important to keep your team informed at all stages and get them bought into the process. That way, when you roll out a change, everyone in your department knows what to expect.

What’s the worst customer service you’ve ever experienced?
I once flew from London to Santiago and got stuck in Toronto with an American airline. They were rude, impatient, and didn’t want anything to do with us! They compensated us with a $35 voucher for food for 24 hours and put us up in a terrible hotel nearby. As if it couldn’t get any worse, they left us stranded on the way back and made us sleep in the airport.

What’s the best thing a customer has ever said to you?
We’re super fortunate that our customers tell us every day how much our app is changing their lives. It’s a really beautiful thing, and to play a small part in helping them is such an honor and why I love doing what I do.

Where do you get your support leadership news?
I’m part of a network called “Customer Support Stories,” which is great for connecting with other like-minded leaders in the industry.

What do you wish people knew about working in customer service?
It’s the bedrock of any successful business and normally a very good reflection of the company’s culture and values.

Conversation closed… for now 😏

If you’re interested in being featured in our Response Time series, you can share your insights on customer service – and what your greatest productivity hack is – with us here.

Intercom vs Zendesk: Two AI agents put to the test

We all know that generative AI is transforming the customer service industry.

AI agents are already handling customer queries with impressive accuracy, and the teams using the right AI solutions are seeing remarkable results. They’re resolving more issues faster and delivering better customer experiences, which is allowing human support agents the freedom to focus on more complex, high-value interactions.

However, identifying the right AI solution is not easy amid all the noise. Our research shows there are significant performance gaps when it comes to resolution rates, accuracy, and quality between different AI agents on the market.

We’ve repeatedly tested our Fin AI Agent against competitors’ offerings to ensure it performs optimally in every way. Here, we’ll show you how Fin compares to Zendesk’s AI agent and walk you through the research process to give you an in-depth understanding of why Fin is the superior choice.

Fin is the best AI agent on the market – with stats to prove it

Let’s start with some numbers.

When we put Fin head-to-head against Zendesk’s AI agent, the difference isn’t just noticeable – it’s remarkable. Three things in particular stood out:

In 80% of cases, Fin provided better answers across the board, demonstrating superior performance in accuracy, completeness, and overall quality. This isn’t just about getting the answers right; it’s about delivering the kind of experience that builds customer trust and loyalty.

“Fin can handle twice the number of complex questions Zendesk’s AI agent can, transforming what’s possible with automated support”

Fin is also much more capable at handling complexity. Unlike Zendesk’s AI agent, which defaults to basic responses when faced with challenging queries, Fin maintains natural conversations by asking clarifying questions. And now, Fin can answer more types of questions with the recent addition of actions. This sophisticated capability means Fin can handle twice the number of complex questions Zendesk’s AI agent can, transforming what’s possible with automated support.

Perhaps most impressively, when dealing with questions that require pulling information from multiple sources – the kind of query that typically needs human intervention – Fin achieves a 96% answer rate, significantly outperforming Zendesk’s 78%. For support teams, this means more queries resolved automatically, faster response times, and happier customers.

These numbers are compelling – but how did we get to them? Here’s an overview of the research process we followed.

How we compared Fin to Zendesk’s AI agent: A look at our evaluation process

Step 1: Setting the stage

To evaluate the AI agents in an unbiased manner, we needed a completely neutral dataset of help articles and relevant questions that we knew were grounded in the articles.

Using ChatGPT 4, we created a fictional bed and breakfast website with 48 comprehensive articles, all of which we loaded into Fin and Zendesk’s AI agent, ensuring a fair playing field.

We also generated 200 customer questions based on the 48 articles. Some were straightforward, while others required piecing together information from multiple articles.

We asked all 200 questions to both Fin and Zendesk’s AI agent.

Step 2: Checking the outputs for hallucinations

Before we started judging the outputs, we checked for any made-up information – hallucinations – in the responses. We found that there was no statistical difference in the hallucination levels between the two AI agents.

Step 3: Judging the answers

We used four advanced AI models (Anthropic Claude 3 Opus, GPT-4, GPT-4 Turbo, and GPT-4 Omni) to act as impartial judges. These “judges” had access to the articles and question bank, and were instructed to vote on the answers provided by both Fin and Zendesk’s AI agent for given questions while considering the articles as the source of truth.

To determine a winner, we applied the Elo rating system, which calculates a score based on which AI agent delivered the better answer, according to the AI judges. Over hundreds of such “competitions,” a clear winner emerged.

The results were clear: when pitted side-by-side, Fin’s answers are almost always better than Zendesk AI agent’s.

Step 4: Digging into the details

We wanted to dig deeper into what specifically made Fin’s answers better. So, we looked more closely at how Fin outperformed Zendesk AI agent in the following areas:

• Providing a direct response.
• Giving the most “readable” answers for humans.
• Delivering a complete resolution of the query.

A direct response

Fin outperformed Zendesk’s AI agent by providing more direct responses to every question type. The most notable difference was in its ability to answer “hard” questions, where Fin answered more than double the questions Zendesk AI agent did, and questions that required piecing together information from multiple sources, where Fin provided answers to 96% of the questions, and Zendesk AI agent only managed 78%.

The most “readable” answers for humans

Accurate answers are one thing, but how they’re structured matter a lot for the end user experience. Fin provided more comprehensive answers than Zendesk’s AI agent, with the average response coming in at 120 words compared to 50 words. Fin’s responses were also formatted to be more scannable, including elements like newlines and bulleted lists.

A complete resolution of the query

Looking at the direct answer results we got, we estimated the probability of a complete resolution provided by Fin by applying the following formula:

In relative terms, Fin was 66% more likely to provide a resolution for a query when both Fin and Zendesk AI agent provided an answer. Similar to the results we saw with the direct response investigation, Fin was also the winner across every answer category.

A few notes on research limitations

While our test was thorough, it had some limitations:

• We used a simulated help center, not real-world data.
• AI judges are great, but they might not perfectly match human judgment.
• The two products we tested have different features, and this could impact the results to an extent.

Overall, these findings clearly demonstrate Fin’s superior performance in direct testing. But beyond the numbers, there are several crucial advantages that make Intercom the clear choice for forward-thinking support teams. Here’s what this means for your business in practical terms.

What sets Intercom’s Fin apart

Flexibility to use Fin as part of Intercom’s seamless AI-first platform – or whatever CS platform you’re currently using

First, we understand that every support team has unique needs. That’s why we’ve made Fin incredibly flexible – you can either use it as part of our comprehensive AI-first system or integrate it with your existing platform, including Zendesk and Salesforce, and access all of its benefits. There’s no need to overhaul your entire support stack or disrupt your team’s workflow – Fin can help you get results in whatever way suits you best.

Pricing that makes sense: 99¢ per resolution

We want AI to be accessible for every team, so we’ve also taken a radically different approach to pricing. While other vendors lock you into complex contracts with hidden costs, we keep it simple: 99¢ per resolution. This transparent, outcome-based model means you only pay for actual value delivered. You don’t have to worry about spending a large chunk on something that doesn’t actually help move your business ahead.

Innovation that keeps you ahead

The thing that has always set Intercom apart is how fast we move. When it comes to AI, staying ahead matters because the sooner you get the latest capabilities, the better your automated customer experience will be.

We’re rolling out new features and capabilities at an unprecedented rate, continuously improving Fin’s performance based on millions of real customer interactions. When you choose Intercom, you’re not just getting today’s best-performing AI – you’re partnering with the most innovative company in the space, ensuring you’ll stay ahead of the curve as AI technology continues to advance.

The future of customer service is here – and it’s already delivering results

Many companies are making noise about their AI capabilities. But what they’re not doing is backing up this noise with evidence. From our research, it’s clear that Intercom’s Fin AI Agent outperforms a significant competitor – Zendesk AI agent – in providing the best, most accurate, high-quality answers. This means you can bring it onboard as part of your team and fully trust in its abilities to resolve a huge share of your customer queries, freeing up your human teammates to focus on more meaningful work.

“In a market full of noise and ambitious claims, we let our results do the talking”

Since this research was conducted, we’re proud to share that we’ve raised the bar even higher by launching Fin 2, our next-generation AI Agent. Delivering human-quality support, it’s capable of achieving a 51% resolution rate straight out of the box, with some of our customers achieving up to 86% after spending some time refining its use. To date, Zendesk is still marketing their first-generation AI agent.

What’s particularly exciting is that this is just the beginning. In a market full of noise and ambitious claims, we let our results do the talking. The data is clear, the performance gaps are real, and the future of customer service is already here. Are you ready to see what Fin can do for your team?

Announcing our latest guide: ‘The New Economics of Customer Service’

Today, we’re excited to share Intercom’s latest guide – The New Economics of Customer Service. In this guide, we unpack how AI enables support teams to offer high-quality support at scale, in an efficient and cost-effective way.

⚡️ Ready to dive straight in and learn how AI broke the linear customer service growth model? Grab a copy →

Growth is a crucial component of any business, and the truth is that sustainable growth is impossible without customer service.

More customers inevitably means higher support volume. And without the resources to handle rising demand for support efficiently, those customers will be left with a poor experience, a bad taste – and a strong desire to take their business elsewhere.

“You can have speed, provide a great customer experience, or keep costs low – choose two”

But scaling customer service alongside business growth has traditionally been a tricky balance to strike. To meet rising demand, the only real option was to add more and more headcount to your support team, which was costly, time-consuming, and unsustainable. This has always been a catch-22 situation for businesses, leading to the all-too-common refrain: you can have speed, provide a great customer experience, or keep costs low – choose two.

AI has changed that; now, there’s another way.

The linear growth model has been broken, which means support leaders no longer have to grow their teams at pace to meet demand. Instead, they can use AI tools to manage increasing support volume quickly and without driving up costs, all while providing a great customer experience.

In other words, it’s now possible to unlock the trifecta of better, faster, and cheaper customer service.

What’s inside

Our new guide reveals how the changing economics of customer service are unlocking new ways for support teams to drive impact and bottom-line results, and offers practical strategies to get set up for success with AI-first customer service.

You’ll learn:

• How to quantify the ROI of AI-first customer service: Understand all the factors you need to consider in order to demonstrate AI’s real value and impact on your bottom line.
• The opportunity cost of not adopting AI: Discover the hidden costs of postponing AI-first customer service – from limited scalability to reduced competitiveness.
• The value-creating opportunities being unlocked for support teams: Explore the changing roles of support agents, and how AI is freeing up time for them to focus on more proactive, revenue-generating, and value-creating work.
• The impact AI is having on support teams, right now: Get real-world advice and strategies from other support leaders for getting started with AI, winning exec buy-in, and driving results.

The economics of customer service have been changed forever

By breaking the linear growth model, AI has created untapped potential for support teams to deliver value to both their customers and their business. At Intercom, we’re saving in the region of $1.75–2 million a year with AI, all while delivering a faster, more personalized support experience for our customers.

These are the new economics of customer service: turning a former cost center into a value driver, and using AI to fuel powerful customer experiences that lead to compounding long-term ROI.

Ready to learn how? Grab a copy of The New Economics of Customer Service here

Braving busy holidays: Reduce customer service stress with automation and AI

The holidays are coming, and for consumer-facing businesses, that often results in a huge tide of shoppers turning to your support team for help. How can you get on top of high conversation volumes, and still provide speedy, personal support during the busiest times of the year?

It’s an easy answer: bring in AI. In the space of a year, we’ve come a long way with this remarkable technology. Providing exceptional (read: personal, efficient) support no longer means needing to add extra headcount or face burnout. Working hand-in-hand with an AI agent, like Fin 2, our latest next-generation release, your human team can now resolve support queries proactively and automatically.

Here are our top tips for making the most of an AI agent like Fin and providing world-class support in a considerably easier way.

Tip 1: Avoid burnout and still provide 24/7 support with an AI agent

No matter how dedicated your support team is, they can’t (and shouldn’t have to) work around the clock during busy holiday periods, such as Black Friday and Cyber Monday. The good news? AI agents like Fin can provide instant responses and accurately resolve queries 24/7.

“AI bots will provide speedy, spot-on answers even when your team is out of office or otherwise engaged”

If you have a knowledge base, you can set up your AI agent and point it at your help content in minutes. Then when a customer asks a question, your bot will provide speedy, spot-on answers to queries even when your team is out of office or otherwise engaged.

The best part is you have 100% control over how you use this automated support. Being able to properly switch off is crucial for avoiding burnout during this time of year, so make use of an AI agent in whatever way best suits your team. Some folks might like to have AI completely take over answering certain questions, while others might like to toggle between automated and human support – with the AI agent being turned on during certain hours and human team members taking over when they’re online. It’s completely up to you.

Tip 2: Automatically answer holiday FAQs in a way that feels personal

Answering the same questions over and over again can feel like running on an endless treadmill for your team. This is especially true during the holiday season when customers have more questions about shipping, delivery times, and holiday discounts and deals.

Luckily, that’s a thing of the past with support from AI agents that can instantly resolve FAQs and reduce your conversation volume. Fin is excellent at this – straight out of the box, it can resolve up to 51% of all queries with 99.9% accuracy. Just in time for the holidays, we’ve put a bow on this level of efficiency and launched a few new sparkly features to ensure Fin not only clears your queue in record time, but makes it a merry experience:

• Fin’s advanced Knowledge Hub helps it maintain up-to-date information about your products and services, ensuring accurate responses throughout the holiday season without requiring constant updates from your team.
• New behavior features allow you to customize Fin’s tone of voice to match your brand’s holiday communication style, choosing from five preset tones or adjusting between concise and conversational responses to maintain consistency across all customer interactions.
• The ability to communicate fluently in 45 languages, ensuring all your customers across the world receive the same high-quality support.

Sounds great, but wondering which questions are best to automate? We recommend digging into data from previous years and identifying patterns in what simple, frequent queries are eating into your team’s bandwidth most.

If you’re looking for inspiration, here’s a list of common questions our customers leave to Fin:

• What are your shipping options, prices, and timelines?
• What is the shipping process for international orders?
• What are your holiday return and exchange policies?
• How do I apply the promo code?
• What’s the deadline for placing an order to guarantee delivery before Christmas?

With Intercom, you can also create Custom Answers to ensure Fin serves the right customers with the right tailored solutions and suggested actions at the right time. For example, if a customer asks about modifying a holiday order, Fin can connect to your order management system, pull in the exact order details, and help them make changes – all within the Messenger.

These kinds of personal touches speed up resolutions, enhance satisfaction, and build strong customer loyalty for your brand.

Tip 3: Set customer expectations around response times

What’s one of the fastest ways to frustrate a customer? Failing to set expectations or deliver on your promises. Again, this is something teams no longer have to worry about, with an AI agent on hand to instantly solve simpler queries while the humans deal with more complex conversations.

“Customers can then plan their next move based on real-time information, instead of waiting around for a response”

If there is any reason that customers might have to wait longer during the holiday rush, build trust and goodwill by proactively communicating response times. Customers can then plan their next move based on real-time information, instead of waiting around for a response that might take a few hours.

With Intercom’s Messenger, you can set clear expectations for holiday shoppers. First, tailor the Messenger intro to share your team’s availability or other important customer service issues. Second, if you’re providing extended holiday hours, you can adjust your team’s office hours and expected response times.

To bring even more visibility to your team’s availability, you can set Fin up to let your customers know what your office hours are and when your team will be back online if they’re looking to speak with a human.

Tip 4: Route complex issues to the right team ASAP

People getting stressed out during the holiday season is inevitable. With all sorts of things happening behind the scenes, it’s important to go into this period with understanding and patience – for your customers and your support team. With that in mind, answers to some questions, such as emotionally charged and complex queries, are probably best handled by a human.

We’re working on a “Category Detection” feature for Fin to help ease the pressure around resolving stressful scenarios quickly. This new feature will analyze language used and automatically identify customer frustration or urgent issues in real-time.

Whether someone is experiencing a technical issue or is understandably angry about a misplaced holiday delivery, Fin can route their conversation to the right team immediately. This intelligent routing works seamlessly across all channels – chat, email, and WhatsApp – ensuring consistent support no matter how customers reach out.

Another helpful thing we did ourselves was set up both Slack and PagerDuty integrations with our Messenger to trigger emergency out of hours notifications for certain scenarios. This ensured we never missed anything critical and could use the most effective option to help our customers, regardless of when they experienced an issue.

Tip 5: Beat your customers to it by being proactive

What’s even better than automatically resolving frequent questions? Proactively solving issues and resolving queries before they ever become problems!

Use banners, automated pop-up messages, or whatever jumps out to your customers on your website or within your product, to provide answers to your top 10 holiday questions. Whether it’s delivery cut-off dates or return policies, make the most important information visible so customers can self-serve.

“By transparently informing your customers of issues, you can help them to make more informed choices and manage their expectations”

It’s also a good idea to use your AI agent or other attention-grabbing features, like a banner or targeted message, to proactively communicate any known issues to customers. By transparently informing your customers of processing times, website bugs, or shipping delays before they make their order, you can help them make more informed choices and avoid unnecessary confusion or frustration.

Tip 6: Empower customers to resolve their own issues

Today’s customers overwhelmingly prefer self-service. So much so, that by 2030, Gartner estimates that a billion service tickets will be raised automatically by customer-led automation. In turn, providing a seamless self-service experience lets you reduce time spent on simple issues and improve your holiday bottom line.

Fin 2 takes self-service to the next level by accessing customer data to provide personalized responses. For example, when a customer asks about their holiday order status, Fin can access their specific order information and provide real-time updates. Using action templates, Fin can even help customers make simple changes to their orders, such as updating shipping addresses or modifying gift options, without requiring human intervention.

As we mentioned before, Fin is fluent in 45 languages. It can automatically detect a customer’s language and serve them relevant answers in that language, providing a great global customer service experience. What’s more, when you integrate third-party apps, like Stripe and Shopify, with Fin, your customers can go beyond just chatting to complete actual transactions.

Tip 7: Proactively help customers break past friction

The average cart abandonment rate in 2024 is a significant 70.19% according to a recent study. Cart abandonment rates typically peak during busy shopping periods, like Black Friday and Cyber Monday. One of the best ways to help customers complete their order is by triggering a targeted, automated proactive message on the checkout page to preemptively answer their questions.

For example, if customers are on the shipping page for a few minutes, they might have questions about your shipping times, returns policy, or something else. You can trigger an outbound message that links to your top FAQs related to purchasing so they get the relevant information upfront without needing to reach out for support.

Stay on top of your holiday customer service

The holidays are a huge opportunity to attract more customers and drive revenue, and in the past that came with extra demands on your support team. Now, AI has truly revolutionized customer service – so much so that you can genuinely handle the holiday season just like any other time of year.

With a helping hand from cutting-edge support tools like Fin, your human team can spend more time focusing on the more meaningful, emotional parts of the job that they enjoy. The dual ability to meet demands quickly and efficiently, and also connect deeply with people when it counts means you’ll win the hearts and minds of customers and they’ll keep choosing your business again and again.

Response Time: Vol. 38

You satisfy your customers, but can you satisfy our curiosity?

With Victor Salinas, Head of Customer Success at VMetrix.

Please tell us a little bit about your company and what you do there.
VMetrix is a software as a service (SaaS) platform designed for financial institutions that allows them to manage investments, risks, and treasury operations in a comprehensive way. It includes everything from financial asset trading to portfolio accounting. I’m Head of Customer Success, and responsible for managing the relationship between our users and the company, for which Intercom has become an essential tool.

What word or phrase in customer service jargon should be retired?
“Unfortunately, this option is not available, but we will strongly consider your suggestion for future versions.”

Which celebrity would be really great at your job, and why?
Ryan Reynolds, because he takes everything with humor.

What’s the most valuable thing that working in customer service has taught you?
You don’t truly understand the customer’s needs until you speak with them every day.

Describe the essence of great customer service using only three words.
Care for them.

Which movie robot would you choose as your AI sidekick?
Baymax from Big Hero 6.

What can you do that a bot will never be able to replicate?
Find alternative solutions within our system so that the user doesn’t have to stop their work.

What’s the most embarrassing thing you’ve ever said/done to a customer?
Responding to a customer with information intended for someone else.

Do you identify more with the title “customer support,” “customer service,” “customer success,” or “customer experience,” and why?
“Customer success,” because I consider myself a partner in their daily growth.

What’s the one piece of advice you would give to your peers in the customer service industry?
Build solutions based on their (the customer’s) needs.

What’s the worst customer service you’ve ever experienced?
When people are more concerned with charging me than with solving my problems.

What’s your greatest productivity hack?
Make lists and prioritize tasks, then set follow-up reminders.

What book are you reading at the moment?
Fall in Love with the Problem, Not the Solution by Uri Levine.

If customer service was an Olympic sport, what would be the main event?
A marathon, because relationships are built over time and in the long run.

What’s the best thing a customer has ever said to you?
“I knew I needed something, but I didn’t know what it was until you helped me understand it.”

What gif best describes your mental state right now?

Where do you get your support leadership news?
From LinkedIn, Intercom’s blog, and through discussions with my team and leaders.

What do you wish people knew about working in customer service?
Two things: collaborating with customer service is an excellent way to improve and build software, and the client doesn’t always know exactly what they need.

If you wrote a book about your experiences in customer service, what would the title be?
“It Takes Two to Dance.”

Conversation closed… for now 😏

If you’re interested in being featured in our Response Time series, you can share your insights on customer service – and what your greatest productivity hack is – with us here.

Response Time: Vol. 37

You satisfy your customers, but can you satisfy our curiosity?

With Kelly Burnette, Classroom Success Manager at Writable from HMH.

Please tell us a little bit about your company and what you do there.
Writable builds lifelong writing and reading skills for students in grades 3-12. We are now a part of HMH, and our team helps drive innovation in education technology through thoughtful and intentional AI tools. As a Classroom Success Manager, it is my job to make sure that educators using our program are supported when they need it most – usually in front of a class full of students!

What’s the most valuable thing that working in customer service has taught you?
The power of listening. Oftentimes, a customer will think their problem is one thing when it is actually something entirely different. By truly listening and knowing my product, I can help resolve the problem at its core and improve their experience.

Describe the essence of great customer service using only three words.
Timely. Clear. Kind.

Which movie robot would you choose as your AI sidekick, and why?
EVE from Wall-E. She knows her mission and isn’t going to back down from it, even when she’s told to. My mission is to help the customer, and sometimes I have to break some rules to do so!

What can you do that a bot will never be able to replicate?
I have a shared experience with our customers, knowing what it’s like to use technology in the classroom as a teacher. No bot can combine my product expertise and personal experience to understand our users’ unique realities.

What’s the most embarrassing thing you’ve ever said/done to a customer?
I’ve mistyped a lot of things because we respond very quickly, but I don’t really get embarrassed! I find my customers usually appreciate the humanity of the interaction.

Do you identify more with the title “customer support,” “customer service,” “customer success,” or “customer experience,” and why?
All of the above! We have a very specific role – “Classroom Success.” That’s because in the world of education, the most important place our app needs to work is in the classroom. Our teachers don’t have time to wait for us to escalate issues or ask other teams, so we are uniquely trained to be tech support, curriculum experts, implementation gurus, and customer advocates all at once.

What’s the one piece of advice you would give to your peers in the customer service industry?
Tag those kind comments for a rainy day. Most customer experiences are positive, but those negative ones can really mess up your attitude. I like to tag my bright spots and appreciative chats so when I’m feeling run down I can go back and remember that I do make a positive difference in my customers’ days.

What’s the worst customer service you’ve ever experienced?
I refuse to call our internet provider – I make my partner do it. The hoops I have to jump through to get to the correct person, and the way they try to manipulate customers into buying higher speed internet when they can’t deliver the quality we already pay for is insane.

What’s your greatest productivity hack?
I block the first 15 minutes of the day to lay out my priorities for this day, this week. I block the last 15 minutes to check on whether I accomplished those things, which gives me a chance to celebrate my wins and feel prepared for what comes next.

What book are you reading at the moment?
I am a fan of fantasy and fiction. I just finished Two Twisted Crowns by Rachel Gillig – it’s a really great duology!

If customer service was an Olympic sport, what would be the main event?
Juggling multiple high priority conversations at once! If a customer asks, “Are you still there?” you are disqualified.

What’s the best thing a customer has ever said to you?
“I am going to go teach all my teammates what you showed me.” Turning customers into champions. 💪

What gif best describes your mental state right now?

Where do you get your support leadership news?
I always attend Intercom’s webinars and check out the resources. I also stay active on LinkedIn and the communities there.

What do you wish people knew about working in customer service?
It’s not as awful as it’s made out to be. Yes, we have our bad days, but you get the chance to connect with a lot of different people. More often than not, our customers are grateful and kind to us, and we get to make a difference in their day. I love getting to turn a negative experience into a positive one for them.

If you wrote a book about your experiences in customer service, what would the title be?
“Is There Anything Else I Can Help You With?”

What’s the strangest thing a customer has asked you?
I once had a customer ask if I could come over to help them figure out how to check their work email at home. They were in Oregon, I’m in Virginia.

What’s your most used emoji in customer chats?
😅

Conversation closed… for now 😏

If you’re interested in being featured in our Response Time series, you can share your insights on customer service – and what Olympic sport customer service would be – with us here.

Fin over email: How we built a multichannel AI agent

Email is an essential channel for support, but email conversations lead to slower resolutions for customers when compared with synchronous channels like live chat.

With the advent of AI-first customer service, a lot of frontline customer queries are now being dealt with by LLM-powered AI agents. Our own Fin AI Agent resolves more than 50% of customer queries immediately.

However, there’s a perception that AI agents can only function over chat. Our research has shown that many customer service leaders continue to equate AI to chat experiences, rather than thinking about how it can deliver support across multiple channels, just as human agents can.

Well, we’re changing that perception with the latest updates to Fin AI Agent – customers can now get instant responses to their emails.

Customers can now get AI answers to their emailed support questions

Getting Fin AI Agent to work over email presented some interesting technical and UX challenges – here, we dive into the process and share some of our learnings.

How Fin for email works

When a user contacts a business’ customer support team via email, Fin AI Agent will automatically jump into the conversation to resolve the issue. Fin’s answers use generative AI technology to create the answer based on a range of support content using the Retrieval-Augmented Generation (RAG) framework.

Fin not only provides direct answers to queries, it’s also more conversational, with the ability to ask clarifying questions if the user’s initial message isn’t clear enough to find the best response. For the most complex cases that Fin isn’t able to answer, Fin will seamlessly hand over to a support agent.

Our development journey

When Intercom launched Fin AI Agent in March 2023, it was the first generative AI-powered customer service agent on the market. We tapped into learnings from our previous machine learning-based product, Resolution Bot, to inform what a generative AI Agent could look like. Since then, we’ve continued to improve and expand our offering by introducing completely new features or rolling out improvements to the underlying model, thereby increasing resolutions.

Starting from first principles

When it came to defining how we would build Fin over email, we didn’t have a blueprint for what the solution should look like. Email as a channel is very different from chat, so we were unsure whether Fin over email should work in the same way. This is where our “Think big, start small, learn fast” principle became relevant, and pushed us to apply first principles thinking.

We started with research to better understand why email automation was important for customers, what kind of requirements they had, and what impact we could anticipate if we built Fin over email. The insights were summarized into a doc called an “Intermission”, which we create at the start of all product initiatives, in keeping with our “Start with the problem” principle.

Iterative development

We decided to start small with an alpha version as there were many assumptions to validate. The team proceeded to build the technical foundations and a very simple teammate experience – just enough to be able to set Fin live on email, but with no bells and whistles. Since we already had a lot of the building blocks in place – a solid email solution and a very flexible automation system (Workflows) – we were up and running quickly.

“This close partnership is at the heart of how we work in R&D – it allows us to move fast as we have tight feedback loops with the customers who will use and benefit from our product”

We reached out to a handful of Fin AI Agent customers, who have a high number of monthly email conversations, to provide us with feedback on what we had built so far. This provided us with enough insight to define scope for our open beta release.

At Intercom, we are very fortunate to be able to partner with our customers as we make progress on our thinking. We work closely together to understand their needs and gather feedback on our initial solution. This close partnership is at the heart of how we work in R&D. It allows us to move fast as we have tight feedback loops with the customers who will use and benefit from our product.

The early feedback helped us shape our open beta. At this stage, we kicked off a more in-depth design phase, resulting in an artifact called an “Interconcept”. This phase of development is driven by the product designer and outlines a set of different approaches, each with a list of pros and cons.

When we were ready to start building, the lead product engineer created a project plan to outline what we needed to build and in what sequence, making it very easy to bring the rest of the team together. Once we launched Fin over email to open beta, we focused on monitoring usage and gathering as much feedback as possible, aiming to uncover any necessary improvements or new functionality required for general availability.

Challenges and considerations

Despite the team working on Fin AI Agent for over a year, amassing deep usage insights and seeing a great deal of success, making Fin work over email came with its own challenges.

Technical challenges

In 2022, prior to the generative AI explosion, Intercom launched the ability to run automations and chatbots over different channels, such as WhatsApp, SMS, and email. At the time, email already proved to be a more complex channel to automate.

From a technical perspective, some examples of challenges we faced when working with email automation were:

• Email deliverability was out of our control – mail clients (such as Gmail and Outlook) can block addresses and throttle usage.
• Multiple queries in the same message happen more often over email, meaning that we needed to ensure we process them separately so no context is lost.
• Converting automated content for chat (which tends to be shorter, separate messages) into a single email with correct formatting (i.e a heading, the body, and an email signature) was not a trivial task.

User experience considerations

Besides the technical challenges, we also had to solve problems that impacted the end user experience.

For most end users, talking to an AI agent over email is a much less established habit than talking to one over live chat, which meant that we had to design an experience that took all the standard expectations around email into account, rather than trying to replicate a chat experience over email.

It was also important for us that the experience felt natural and intuitive so that end users felt comfortable with interacting with an AI agent over email.

We had to consider many differences between live chat and email when designing the new experience, such as:

• As email is an asynchronous channel, conversations don’t have an instant back-and-forth like they do over live chat and customers often have to wait longer to receive a response to their question.
• The email content is usually longer and contains more information, whether that’s text or images.
• Interactive steps that you can add to chat conversations, such as buttons, don’t quite translate over to email.
• Setting expectations that a user is talking to an AI agent requires different visual cues in email than over live chat.
• Emails render very differently across a number of email clients (e.g. Gmail and Outlook), resulting in a long list of design requirements.

Adapting our underlying AI architecture

Lastly, with the learnings gathered from both the technical and end user experience challenges, we partnered with Machine Learning scientists and engineers to create a new component in the AI agent’s underlying architecture specifically for email. Different to our original AI agent over chat, this new agent was developed with the specificities of email in mind, such as:

• Ability to process multiple questions from a single message separately; for example, it can directly answer some queries and clarify others in the same email response.
• Not processing email signatures containing images that are not relevant to the query.
• A built-in mechanism to ignore spam and automated emails.

As the expectation for a response over email isn’t as instantaneous as chat, we were also able to perform some more complex LLM querying for better and more robust answers without significantly impacting the response times.

Fin over email in action

The impact for our customers has been immediate. For instance, Robb Clarke, Head of Technical Operations at RB2B, reported these astonishing results:

“RB2B 2x’d its user base in the last 58 days but my support team is fielding 45% LESS inquiries thanks to one major change, Fin AI Agent started handling email replies. This simple yet powerful change saved us from handling an additional 493 tickets. At 15 minutes per ticket, that’s about 123 hours saved. If you’re not using it yet, you’re missing out. The efficiency and time savings are game-changers – 12 months from now, our team of 2 is going to be acting like a team of 20.”

Within the first month of release, Fin processed over 1 million end user emails. Fin has provided an AI-generated answer to over 81% of the email conversations it has been involved in, automatically resolving more than 56% of them on average.

Fin over email is available now. Learn more about how it can transform your customer support experience, or check out this instructional video, which shows you how to set it up to support your customers.

Your customer service experience has to deliver great support everywhere your customers expect to communicate with you, and that means AI agents have to be able to deliver support in those channels too. With Fin AI Agent, that omnichannel AI-powered support experience is a reality.

Evolving Intercom’s database infrastructure

Intercom is rolling out a major evolution of our database architecture moving to Vitess – a scalable, open-source MySQL clustering system – managed by PlanetScale.

For many years, Intercom has used Amazon Aurora MySQL as our default database. With the addition of our custom sharding solution for high scale data, Aurora MySQL has allowed us to scale our databases with relative ease. It has supported hundreds of terabytes (TB) of data, 1.5 million reads per second, and tens of thousands of writes per second. Aurora MySQL has served us well as the source of truth for the majority of Intercom’s most critical production data.

“We deeply understand the importance of reliability because we experience it firsthand”

For our customers, when Intercom is down, critical parts of their business are affected. They expect flawless uptime, and so do we, even accounting for unforeseen disruptions or planned database maintenance. Our own teams – including Customer Support, Sales, Product, Engineering, IT, and more – rely heavily on our platform every day. An outage doesn’t just impact our customers; it impacts us directly. We deeply understand the importance of reliability because we experience it firsthand.

In late 2023, as we reviewed our database architecture, several factors led us to seek improvements: enhancing the customer experience, addressing operational friction, and keeping pace with a shifting database landscape.

Our review surfaced these goals:

• Eliminate downtime due to database maintenance and writer failovers.
• Reduce the complexity and cognitive load of working with databases across engineering teams.
• Streamline the migration process and improve the latency of running large-scale database table migrations.
• Achieve straightforward, low-effort scaling of MySQL for the next decade.

We aim to build “boring” software and are committed to running less software, choosing to build on standard technologies, and outsource the undifferentiated heavy lifting. With this in mind, we decided earlier this year to move our database layer to Vitess managed by PlanetScale within our AWS production accounts.

Why Vitess?

Vitess is a MySQL-protocol compatible proxy and control plane for implementing horizontal sharding and cluster management on top of MySQL. Originally developed by YouTube and now used by companies such as Etsy, Shopify, Slack, and Square, Vitess combines MySQL features with the scalability of NoSQL databases. It offers built-in sharding capabilities that enable database growth without necessitating custom sharding logic in the application.

Vitess automates tasks that impact database performance, such as query rewriting and caching, and efficiently handles functions like failovers and backups using a topology server (a system that keeps track of all the nodes in the cluster) for server management. It addresses the lack of native sharding support in MySQL, facilitating live resharding with minimal downtime and maintaining up-to-date, consistent metadata about cluster configurations. Importantly, it also acts as a connection proxy layer which would eliminate the majority of database related incidents we’ve had in recent years. These features effectively provide unlimited MySQL scaling.

Why PlanetScale?

PlanetScale builds upon Vitess by offering a managed platform that provides an exceptional developer experience and handles the undifferentiated heavy lifting of managing the underlying infrastructure. Their expertise, which includes core Vitess team members, allows us to benefit from advanced features like advanced schema management, database branching, and automated performance optimization.

The details around scale and challenges below largely relate to our US hosted region – the infrastructure in our European and Australian regions is similar but at a smaller scale. PlanetScale will be rolled out to all regions.

Supporting high scale: 2011 to 2024

As Intercom scaled, we adapted our database strategies in three main ways:

• Get a bigger box: In the very early days of Intercom, scaling our databases was straightforward – we simply upgraded to larger and more powerful database instances. This vertical scaling approach allowed us to handle increased load by leveraging AWS’s flexible and ever improving instance types. With a maintenance window, we could move to instances with more CPU, memory, and I/O capacity as our data and traffic grew. However, this strategy has its limits. There’s only so much capacity you can add before hitting the ceiling of what a single machine can handle, both in terms of hardware limitations and ability to perform certain operations such as database migrations.
• Functional sharding: To move beyond the constraints of vertical scaling, from 2014 we started implementing functional sharding within our architecture. This involved splitting our monolithic database into multiple databases, each dedicated to specific functional areas of our application. For example, we separated our conversations table out into its own database. By distributing the load across dedicated databases, we reduced contention and improved performance for specific workloads. This has its drawbacks, cross-database queries became more complicated, and maintaining data consistency across different shards required additional coordination through multi-database transactions. As AWS introduced larger and more powerful database instances, this scaling strategy has remained relevant.
• Move to RDS Aurora: Soon after AWS released RDS Aurora in 2015, we eagerly migrated to RDS Aurora from the original RDS MySQL offering. Aurora’s architecture decoupled storage from compute, and allowed us to easily scale-out using read-replicas, avoiding replication lag and other problems that existed in traditional MySQL implementations at the time.

Sharding per customer

As our customer base and data continued to expand significantly, we faced database scalability challenges that could no longer be addressed by vertical scaling or functional sharding. To overcome this, we implemented customer sharding by horizontally partitioning our data based on customer identifiers. This approach allowed us to distribute the load more evenly across multiple database clusters and scale horizontally further by adding new database clusters as needed. Effectively, each customer would have their own database for high scale data (e.g. conversations, comments, etc.).

“Our sharding solution enabled us to handle billions of data rows and millions of reads and writes per second without compromising performance”

Building our own sharding solution was a substantial undertaking which we completed in 2020. We dedicated a team to develop a tailored solution using technologies we were already familiar with. This enabled us to handle billions of data rows and millions of reads and writes per second without compromising performance. Thanks to this setup, we were now able to migrate large-scale tables that we hadn’t been able to touch for years, unlocking easier and faster feature development.

Managing this sharded environment introduced new complexities. For example, our application had to incorporate logic to route queries to the correct shard and simple migrations, for example adding a new table, would take days to complete. This was better than not being able to change these tables at all, but still not optimal.

What problems did we see in our current setup?

Connection management

Intercom operates a Ruby on Rails application with its primary datastore being MySQL. In the USA hosting region, where the vast majority of Intercom workspaces are hosted, we run 13 distinct AWS RDS Aurora MySQL clusters.

A problem of this architecture is connection management to MySQL databases. There are limits on the maximum number of connections that can be opened to any individual MySQL host, and on Amazon Aurora MySQL the limit is 16,000 connections. Intercom runs a monolithic Ruby on Rails application, with hundreds of distinct workloads running in the same application across thousands of instances, connecting to the same databases.

“The use of ProxySQL allows us to scale our application without running into connection limits of the RDS Aurora MySQL databases”

As each running Ruby on Rails process generally needs to connect to each database cluster, the connection limit is something we had to engineer a solution for. On most of the MySQL clusters, the read traffic is sent by the Ruby on Rails application to read-replicas, which spreads the connections out over a number of hosts, in addition to horizontally scaling the query load balancing across the read-replicas.

However, for write requests, we need to use a different approach, and in 2017 we rolled out ProxySQL to put in front of the primary writer nodes in each MySQL cluster. ProxySQL maintains a connection pool to each writer in the MySQL clusters and efficiently re-uses connections to serve write requests made by our Ruby on Rails application. The use of ProxySQL allows us to scale our application without running into connection limits of the RDS Aurora MySQL databases.

In the last year, we’ve experienced a number of outages related to our use of ProxySQL. These issues arose particularly when we attempted to upgrade to ProxySQL 2.x and utilize new features like its integration with RDS Aurora read replicas, which led to instability and outages.

Database maintenance

Maintenance windows are a necessary evil of most database architectures, and nobody loves them. For many of our customers, when Intercom is down, large parts of their business is down too. This is increasingly relevant as Intercom builds out features such as Fin AI bot, which can resolve large amounts of conversations for our customers.

Maintenance windows are something we’ve avoided unless absolutely necessary and when needed, run the majority of them at the weekend in order to reduce the impact for our customers. With AWS Aurora, any upgrades or planned instance failovers (for example, for increasing the size of a database instance) required maintenance windows with customer impact ranging from five to seventy minutes.

For instance, during our upgrade from Aurora 1 to Aurora 2, we conducted ten maintenance windows across our regions, each causing actual disruptions between twenty and seventy minutes.

We knew we needed to do better here, and remove the need for maintenance windows entirely.

Intercom’s database architecture 2024 and beyond – enter PlanetScale

While these methods have allowed us to scale with relative ease, the database landscape has changed dramatically. Compared to 2019, when we decided on our custom application sharding approach, there are now more options for building practically infinitely scalable databases appropriate for Intercom.

Embracing Vitess and PlanetScale

To address the limitations and complexities of our existing database architecture, we have embarked on a journey to adopt Vitess managed by PlanetScale. This transition represents a significant evolution in our approach to database management, aiming to enhance scalability, reduce operational overhead, and improve overall availability for our customers. We have already migrated several databases and have many more to transition in the coming months. The benefits we’re already seeing include:

Simplifying connection management

One of the immediate benefits of Vitess is its ability to act as a single connection proxy layer through its VTGate component. VTGate is a stateless proxy server that handles all incoming database queries from the application layer. It intelligently manages connection pooling and query routing, effectively multiplexing a large number of client connections over a smaller number of backend connections to the MySQL servers.

“VTGate allows us to scale our application seamlessly without worrying about connection constraints”

By centralizing connection management, VTGate eliminates the 16,000 connection limit per MySQL host that we previously faced with Aurora. This removes the need for ProxySQL in our architecture, reducing a massive source of complexity, and potential points of failure. VTGate also provides advanced query parsing and can route queries based on the sharding key or even handle scatter-gather queries across multiple shards when necessary. This allows us to scale our application seamlessly without worrying about connection constraints or overloading individual database instances.

Zero-downtime maintenance and failovers

Vitess offers advanced features like seamless failovers, which are critical for eliminating customer downtime during maintenance operations such as software upgrades and changing instance sizes. Its built-in failover mechanisms ensure that if a primary node goes down, a replica can take over almost instantaneously without impacting ongoing transactions. This aligns perfectly with our goal of providing flawless uptime and eliminates the need for extended maintenance windows that disrupt our customers’ operations. With the clusters we’ve already migrated, we can refresh the database instances without any noticeable impact on our customer-serving metrics.

Native Sharding Support

Perhaps the most significant advantage of Vitess is its native support for horizontal sharding. Unlike our previous custom sharding solution, Vitess abstracts the complexity of sharding away from the application layer. Our engineers no longer need to write custom logic to route queries to the correct shard; Vitess handles it automatically based on the sharding scheme we define.

“This reduction in cognitive load allows our teams to focus more on delivering new features and less on managing database intricacies”

In time, we will also be able to combine our functionally sharded databases into a single logical database thereby reducing the complexity we introduced to maintain data consistency across the databases. For example, currently, if a new comment is created, three individual databases must be kept in sync. This reduction in cognitive load allows our teams to focus more on delivering new features and less on managing database intricacies.

Streamlined migrations and scalability

Running large-scale database migrations has been a pain point due to the time and complexity involved. Migrations on our largest non-sharded tables can take months to complete. Vitess addresses this with its online schema change tools operating on sharded data, enabling us to perform migrations with minimal impact on performance. Additionally, scaling horizontally becomes a straightforward process. Need more capacity? Simply add new shards, and Vitess will manage the data distribution without requiring significant changes to the application.

Partnering with PlanetScale

By choosing PlanetScale to manage our Vitess deployment within our AWS production accounts, we leverage their expertise and the contributions of the Vitess core team members they employ. PlanetScale provides a developer-friendly experience and takes on the undifferentiated heavy lifting of managing the underlying infrastructure. This partnership ensures that we benefit from best-in-class database management practices while allowing us to remain focused on what we do best: building our AI-first customer service platform for our customers.

One of the standout features PlanetScale offers is its advanced schema management capabilities. PlanetScale enables non-blocking schema changes through a workflow that allows developers to create, test, and deploy schema modifications without impacting the production environment. This is facilitated by their concept of database branching, akin to version control systems like Git. Developers can spin up isolated database branches to experiment with changes, run tests, and then merge those changes back into the main branch seamlessly. This drastically reduces the risk associated with schema migrations and empowers our engineers to iterate faster, ultimately accelerating our product development cycles. Just like with Git, if a database schema change is pushed to production and an issue is discovered, it can be reverted easily.

“This new mechanism improved the latency of the previously expensive query by 90%”

PlanetScale also allows for net new mechanisms we can use to serve requests. For instance, we recently used materialized views to optimize the counting of open, closed, and snoozed conversations for teammates. This new mechanism improved the latency of the previously expensive query by 90%, leading to a faster teammate experience and reduced database load.

Additionally, PlanetScale provides automated index and query optimization tools. The platform can analyze query performance and suggest or automatically implement index improvements to enhance database efficiency. This proactive approach to optimization reduces the operational overhead typically associated with manual database tuning – everyone on the team can now operate like a world class database expert. These improvements ensure that our queries run efficiently and allow us to maintain high application performance, which translates to a smoother and more responsive experience for our customers.

Challenges faced during migration

Moving the databases that are responsible for Intercom’s most critical data is a major undertaking and it has not been without its challenges. Despite thorough planning and testing, we encountered several issues that provided valuable learning opportunities and ultimately strengthened our migration strategy as we move across more databases.

Latency spikes due to cold buffer pools

One of the initial hurdles was unexpected latency during the cutover of one of our core databases to PlanetScale. When we redirected traffic to the new Vitess cluster, we anticipated some initial latency as the database caches warmed up. However, the latency spikes were more significant and lasted longer than expected – particularly in one availability zone.

This was primarily due to cold buffer pools on the MySQL instances within Vitess. Since these instances had not served production traffic before, their caches were empty. As a result, queries that would typically be served from memory had to fetch data from disk, increasing response times. While we anticipated this problem, we expected only a few seconds of latency, however in reality it continued for twenty minutes and made the Inbox slow to respond to customer requests.

To mitigate this for subsequent migrations we’ve implemented read traffic mirroring to pre-warm the buffer pools before redirecting live traffic. By simulating traffic to load frequently accessed data into memory, we can reduce the initial latency spikes during future migrations.

Disk I/O saturation and resource limits

During periods of high load after the initial cutover and at traffic peak, we observed that some replica servers were experiencing disk I/O saturation. The replicas reached the maximum IOPS allowed by their attached storage volumes. This led to increased CPU utilization in the “iowait” state, further degrading performance.

“Scaling down by removing excess capacity is significantly faster and less disruptive than scaling up under pressure”

The root cause was that the replicas’ IOPS were under-provisioned for the workload they needed to handle. To resolve this, we initiated the scaling out of additional replicas. However, adding new replicas was time-consuming due to the size of our data – restoring backups to new instances and allowing them to catch up with replication took several hours. During this period, standard operations in the Inbox were 1.5 to 3x slower than usual, with Workload Management most affected – slowing to between 5x and 10x normal latencies.

Our takeaway from this is that we will significantly overscale all clusters as we move across load. Scaling down by removing excess capacity is significantly faster and less disruptive than scaling up under pressure.

Configuration changes and unexpected interactions

We also faced challenges when certain configuration changes interacted poorly with application behavior. For instance, increasing the transaction pool size and the maximum transaction duration seemed beneficial in isolation. However, combined with a surge of scheduled operations, for example bulk unsnoozing of conversations on the hour, these changes led to resource saturation. The database was flooded with long-running transactions, causing latency and errors impacting the Inbox.

The road ahead

Our migration to Vitess is more than just a technological upgrade; it’s a strategic move to future-proof our database architecture for the next decade and beyond. By embracing Vitess and partnering with PlanetScale, we’ve positioned ourselves to provide even greater reliability, scalability, and performance for our customers.

“The lessons we’ve learned and the mitigations we’ve implemented have set us up for success as we continue migrating our remaining infrastructure”

So far, we’ve successfully migrated our databases related to our AI infrastructure and one of our most critical databases powering the Inbox. These early migrations have validated our decision and provided invaluable insights. The lessons we’ve learned and the mitigations we’ve implemented have set us up for success as we continue migrating our remaining infrastructure.

Looking ahead, we’re excited about the possibilities that Vitess and PlanetScale open up for us. The native sharding capabilities will allow us to simplify our database architecture, reducing complexity and operational overhead. Our teams can focus more on delivering innovative features and less on managing database intricacies, ultimately enhancing the experience for our customers.