Monitors help you continuously evaluate and improve Fin’s conversation quality at scale.

They give you a structured way to define which conversations should be reviewed—whether that’s a random sample for baseline quality, or a targeted set based on higher-risk or higher-impact signals. This replaces ad-hoc sampling and spreadsheet-driven QA with a repeatable system that scales as volume grows.

Monitors work with Custom Scorecards:

Scorecards can include criteria that are:

This ensures quality is assessed consistently, while still allowing flexibility in how reviews are performed.

Teams use Monitors to maintain ongoing visibility into quality and focus attention where it matters most. Common use cases include:

Monitors make it easier to detect patterns, surface issues earlier, and generate insights that can be shared with product, support, or leadership teams.

To access Monitors, go to Fin AI Agent > Analyze > Monitors.

To create a new Monitor, click + Monitor. Pick one of the templates or Start from scratch.

Give your Monitor a name and choose which conversations the Monitor should review.

This can be:

You can narrow down conversations of interest by:

Choose when the monitor should begin evaluating conversations. This allows you to run QA on historical conversations from a specific point in time, as well as continuously surface new matching conversations.

You can control when a conversation is matched to a monitor. This determines when the monitor evaluates the conversation — and, if a scorecard is attached, when that scorecard runs. Select one of the following options:

Use this setting to align evaluation timing with your workflow — whether you want to assess Fin immediately after it finishes, or only once the conversation is officially closed.

All conversations that match the Monitor are automatically assigned to that reviewer, so reviews are routed consistently without manual coordination.

In this example, we're creating a Monitor that flags conversations with "Vulnerable customers", starts finding matches from Now (today), and assigns them to Alissa for review.

You can associate a scorecard with a monitor to automatically evaluate every matched conversation against defined criteria. Once selected, the scorecard runs as soon as the conversation is added to the monitor, and results appear in the monitor for reporting and review.

For monitors that use natural language flag criteria, use the Test monitor tool to validate your criteria against real conversations before you create or update the monitor. It shows which conversations would be flagged and highlights mismatches so you can refine the wording and reduce false positives or misses.

In the Flag criteria section, click Run test or click the Test button on the top right

Review sample conversations

For existing monitors, this list is automatically populated with recent conversations that were flagged and not flagged by the monitor. You can also paste additional conversation URLs or IDs to test specific edge cases.

Check the results

For each conversation, review the Monitor result (Flagged / Not flagged) and mark whether it’s Correct. The evaluation summary shows your overall pass rate and highlights mismatches.

Refine and retest

Update the Flag criteria description and rerun the test until the results accurately reflect what you want the monitor to capture.

Once the Monitor has been created, it will start finding matches and appear on your Monitors page. You can always edit your Monitor configuration later, if needed.

A custom scorecard defines what “good” looks like for your team by explicitly setting the criteria you care about—such as accuracy, tone, or policy adherence.

You can have multiple scorecards for different Monitors. Simply choose which scorecard you want to associate with a Monitor from the Monitor set up screen:

Go to the Fin AI Agent > Analyze > Monitors. Click Scorecards. You can use the out-of-the box Fin Quality Scorecard or create your own by clicking +New scorecard:

Start by adding scorecard attributes. Click +New attribute

When creating a new attribute, you’ll:

1. Name the attribute. Give the attribute a short, clear name (for example, Sentiment or Answer accuracy). This name appears in reports and will be used as a reference.

2. Describe what’s being evaluated. Add a clear description explaining:

3. Choose how the attribute is scored. Decide whether the attribute should be:

You can mix AI-scored and human-scored attributes within the same scorecard.

4. Define rating options. Add the possible rating values a reviewer or AI can select (for example: Good, Okay, Poor). Each attribute must have at least two rating options. For each rating option, you’ll:

The score you assign determines how that rating contributes to the overall review score.

5. Choose whether to include it in the review score

You can toggle Include in review score on or off.

In this example, we've created a scorecard attribute that evaluates escalation ease:

After adding scorecard attributes, you can configure how they affect the overall review result.

Marking a scorecard attribute as critical

You can mark an attribute as Critical.

If a critical attribute receives a failing rating, the entire review fails:

Critical attributes are useful for non-negotiable standards such as:

Scorecard attribute weighting

Each attribute can be assigned a weight to define its relative importance.

Weights only apply to attributes that are included in the review score. Use weights to reflect what matters most. For example, you might assign a higher weight to Accuracy than to Efficiency if correctness is more important than speed.

Adding a pass threshold

You can define a pass threshold for the scorecard. The pass threshold determines the minimum overall score required for a review to be considered passing. For example:

This is evaluated after weighted scoring, provided no critical attribute has already failed the review.

Once reviews are completed, scores are visible in both the conversation list and within each conversation.

In Monitor, the conversation list shows the overall review score (percentage or Fail) alongside the individual attribute ratings as columns. This makes it easy to quickly scan performance across conversations and spot failures or low scores.

When you open a conversation and go to the Score tab, you can see the assigned scorecard, review status, overall score, and the selected rating for each attribute. This view shows exactly how the final score was determined.

Each monitor gives you a clear view of the conversations it has matched and the scorecard scores. This makes it easy to move from detection to review to action, without leaving Intercom. Click on a Monitor and you can see:

Manual reviews can be completed directly from the Monitor conversation view by clicking on the conversation and filling in the scorecard. AI‑generated scores can also be overridden by human reviewers if needed.

Start with the core principle: Attributes compete. The AI looks at the full list and selects the single best match for each attribute. Your job is to make that choice obvious.

Descriptions carry most of the classification signal.

The description should make it easy for the AI to recognize real-world phrasing, not just abstract definitions.

Attributes within the same scorecard should not compete conceptually.

It’s fine if a single conversation fits multiple attributes across the scorecard. What matters is that within each attribute set, the values are clearly separable.

When reviewing your taxonomy, assess each attribute on:

This structured review forces you to tighten definitions and reduce ambiguity — which directly improves classification performance.

Monitors do not compete. Each monitor runs independently as a yes/no check. Multiple monitors can flag the same conversation — and that’s fine.

Because of this, precision matters more than distinction.

The AI performs better when evaluating explicit signals rather than emotional interpretations.

Reducing false positives is critical for monitors.

Example: “Customer uses profanity. EXCLUDE: mild language such as ‘damn’ or ‘crap.’” If something should not trigger the monitor, say so clearly.

If your monitor depends on sequence or pattern, structure it clearly:

This makes the logic deterministic and easier to evaluate.

We’re expanding Monitors with more powerful ways to detect issues, measure quality, and take action. Upcoming improvements include: