{"id":31891,"date":"2026-03-25T17:50:35","date_gmt":"2026-03-25T17:50:35","guid":{"rendered":"https:\/\/www.intercom.com\/blog\/?p=31891"},"modified":"2026-03-25T17:59:08","modified_gmt":"2026-03-25T17:59:08","slug":"announcing-monitors-opening-the-ai-black-box","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/announcing-monitors-opening-the-ai-black-box\/","title":{"rendered":"Announcing Monitors: Opening the AI black box"},"content":{"rendered":"

At Fin Labs Paris<\/a>, we announced Monitors, a new product area for Fin that sits alongside Insights and Recommendations to give you a full observability suite \u2013 and confidence in what Fin is doing.<\/p>\n

With Monitors, you define what conversations get reviewed, both Fin and human, and set evaluation criteria using Custom Scorecards. This ensures you\u2019re monitoring metrics that matter the most to your business, giving you complete control over support quality.<\/p>\n

When used alongside Insights and Recommendations<\/a>, you have everything you need to see what’s happening across your support operation, hold every conversation to your bar, and continuously drive toward perfect customer experiences.<\/p>\n

As Agents become more powerful, transparency and control become critical<\/h2>\n

AI is getting more capable, fast. Everyone working in this industry feels it.<\/p>\n

We now have Agents executing complex tasks, like handling real customer conversations with real consequences. Fin has almost 8,000 customers, averages a 67% resolution rate, and resolves close to 2 million customer queries every single week, including highly complex queries in regulated industries.<\/p>\n

\"Fin's<\/p>\n

As Agents take on deeper work at that scale, observability is essential. Most support leaders today can’t confidently answer basic questions about what their Agent is actually doing: whether it’s delivering good experiences, resolving complex issues completely, representing their brand the way a human would, and doing all of it reliably, conversation after conversation. The old infrastructure can\u2019t help them, CSAT scores and QA samples have always been limited in scope, they don\u2019t scale effectively.<\/p>\n

The result is a black box.<\/p>\n

What teams need most right now is confidence. That\u2019s why we designed Fin with transparency at its center. Because teams should feel empowered to understand and optimize it on their own.<\/p>\n

At Intercom, this is called the Fin Flywheel: Train, Test, Deploy, Analyze.<\/p>\n

\"Fin<\/p>\n

Analyze is the step where you find out what’s actually happening and it’s where improvement begins.<\/p>\n

We believe that having confidence in your AI operation requires three things:<\/p>\n

    \n
  1. A complete understanding of what Fin, your human team, and your customers are talking about.<\/li>\n
  2. A way to monitor and score conversations based on the criteria that matter most to your business.<\/li>\n
  3. AI-powered recommendations that make it easy to act on what you find.<\/li>\n<\/ol>\n

    To address these, last year we launched Insights<\/a> and Recommendations<\/a>. Now, we’re announcing Monitors \u2013 completing a system for full observability, and the key to opening the black box.<\/p>\n

    Monitors: Know whether every conversation met your standards<\/h2>\n

    Knowing how a customer felt about a conversation is different from knowing whether it was handled correctly; both are important to service quality.<\/p>\n

    Monitors is a new QA capability that delivers a structured, repeatable way to define which conversations get reviewed \u2014 and evaluate them against quality criteria you set. It replaces ad-hoc sampling and spreadsheet-driven QA with a system that scales as your volume grows.<\/p>\n

    Two components work together: Monitors define what gets reviewed and Custom Scorecards define how each conversation is evaluated.<\/p>\n

    The right conversations, and the right coverage<\/h3>\n

    Random sampling was always a blunt tool. When AI is handling thousands of conversations a week, a small, arbitrary slice won\u2019t reliably capture your highest-risk edge cases, your most complex escalations, or where quality is starting to drift.<\/p>\n

    With Monitors, you define how conversations are selected and evaluated. That can mean targeting specific signals of risk or failure, like \u201cthe customer showed signs of financial vulnerability\u201d or \u201cFin looped around with the same answer without resolving the issue.\u201d Or it can mean creating consistent, repeatable generic samples to benchmark quality over time. You set this criteria from an existing list of filters, based on customer data, channel, or Fin-specific metrics, or use natural language to describe instances with more nuance.<\/p>\n

    \"A<\/p>\n

    You can combine both approaches: hone in on the conversations that matter most <\/span>and<\/span><\/i> maintain a steady, structured QA sample each week.<\/span><\/p>\n

    \""When<\/p>\n

    Custom Scorecards: Enforce your standards, consistently<\/h3>\n

    Every business operates differently, so a one-size-fits-all quality rubric won\u2019t reflect your priorities, your trade-offs, or what your customers actually care about.<\/p>\n

    Custom Scorecards let you define what \u201cgood\u201d looks like for your business and turn that into a custom quality score for every conversation.<\/p>\n

    You define the criteria that matters, how each should be measured, and how important each one is. Some criteria can be scored automatically by AI, others reviewed by a human, or both \u2014 all within the same scorecard. This means you\u2019re not choosing between scale and judgment; you get both in one system.<\/p>\n

    \"A<\/p>\n

    Each conversation is then evaluated against these criteria, and the system calculates an overall quality score based on your configuration. You can weigh what matters most, or mark certain criteria as critical, so a single failure can fail the entire evaluation when needed.<\/p>\n

    The result is a single, consistent quality score that reflects your standards \u2014 not a generic metric, and not a collection of disconnected checks. This is what makes quality measurable over time. You can track how your AI and human support are performing against the same definition of \u201cgood,\u201d and see where things are improving or breaking down.<\/p>\n

    There\u2019s an important distinction here: CX Score<\/a> tells you how customers felt about a conversation. Custom Scorecards tell you whether it met your standards. You need both.<\/p>\n

    \""We<\/p>\n

    Review Queue: Turn flags into fixes<\/h3>\n

    When conversations are flagged as needing a human review, based on your criteria, they are placed in the Review Queue. Here, every conversation matched by a Monitor is automatically assigned to the right reviewer, with its scorecard attached and review status tracked:<\/p>\n