Monitors for QAing human teammates let you evaluate and improve the quality of your teammates' conversations at scale. You define which conversations get reviewed, attach a custom scorecard to score them, and assign reviewers - all automatically.

A Monitor selects which conversations to review. A Scorecard defines how each one is evaluated — the specific criteria you care about, such as accuracy, tone, or policy adherence. Together, they give you a consistent, scalable way to measure and improve conversation quality across your team.

Setting up human agent QA takes two steps:

Go to Fin AI Agent > Analyze > Monitors and click Scorecards.

Create your own by clicking + New scorecard:

Start by selecting who's evaluated. Click Human teammates assigned and choose either teammates in specific teams or individual teammates.

Next, add your scorecard criteria. First, click + Criteria > Create new.

When creating a new criteria, work through the following steps:

Give the criteria a short, clear name (for example, Sentiment or Answer accuracy). This name appears in reports and will be used as a reference.

Add a clear description explaining what the criteria checks and how it should be scored. The description is the prompt the AI uses to score this criteria, and the more precise it is, the more accurately AI will evaluate conversations. It also helps human reviewers apply the same criteria consistently.

Decide whether the criteria should be automatically scored with AI, or manually scored by human reviewers. You can mix AI-scored and human-scored criteria within the same scorecard.

Add the possible rating values a reviewer or AI can select (for example: Good, Okay, Poor). Each criteria must have at least two rating options. For each rating option, you will:

The score you assign determines how that rating contributes to the overall review score.

For each rating option, you can define a list of rating reasons which are predefined labels that explain why a particular score was given. Rating reasons help reviewers and AI categorize scores consistently, making it easier to identify patterns across conversations.

When AI scores a criteria, it automatically selects the most relevant predefined reason where one applies. If no predefined reason fits, the AI generates a clear explanation so that every score has meaningful context.

You can automate the entire QA process for a scorecard by toggling on Auto-review scorecard.

When enabled:

After adding scorecard criteria, configure how they affect the overall review result.

You can mark criteria as Critical. If a critical criteria receives a failing rating, the entire review is marked as failed — regardless of how the other criteria scored:

Critical criteria are useful for non-negotiable standards such as compliance requirements, safety or policy adherence, and escalation handling.

Each criteria can be assigned a weight to define its relative importance.

Weights only apply to criteria included in the review score. Use weights to reflect what matters most — for example, a higher weight on Accuracy than Efficiency if correctness is more important than speed.

You can define a pass threshold — the minimum overall score required for a review to be considered passing. For example, if the pass threshold is 80%, any review scoring below 80% is marked as failed.

This is evaluated after weighted scoring, provided no critical criteria has already failed the review.

Here's an example of how three criteria combine into a final score:

Overall score = (100x60 + 50x30 + 100x10) / (60+30+10) = 85%

Monitors define which conversations get reviewed. You set the criteria, choose the reviewer, and attach a scorecard to evaluate quality. Once live, Monitors run automatically and surface matching conversations for your team to action.

You need at least one scorecard before you can get the most out of a monitor that evaluates human conversations.

To access Monitors, go to Fin AI Agent > Analyze > Monitors. Click + Monitor to get started. You can also choose a template for Fin monitors, Teammate monitors, or General monitors.

Give your Monitor a name, then choose how conversations are evaluated. This is where you can link the scorecard you created to evaluate human agents:

Associate a scorecard with the Monitor to automatically evaluate every matched conversation against defined criteria. Once selected, the scorecard runs as soon as a conversation is added to the Monitor, and results appear in the Monitor for reporting and review.

This is also where you can select your reviewers. All conversations that match the Monitor are automatically assigned to the selected reviewers, so reviews are routed consistently without manual coordination.

Your Monitor can target:

You can narrow down conversations using:

Select how the Monitor runs:

Choose when the Monitor should begin evaluating conversations. This lets you run QA on historical conversations from a specific point in time, while continuously surfacing new matching conversations from that date forward.

A conversation must be closed before it can be evaluated by a human agent QA monitor.

Conversations can be reviewed and submitted from various views.

In all views:

There are several ways to access and submit reviews. Click a monitor to view all reviews associated with it or on the Inbox page click on Assigned to me to directly view all the reviews you're in charge of.

or click on Reviews received to directly view all the reviews that have been submitted for you as the teammate being reviewed.

To complete a review:

Monitor reports help you track and measure conversation quality. You can use these metrics to build reports that highlight quality trends and identify areas for improvement.

All Monitor metrics are available in the custom report builder, so you can combine them with other Intercom data to create tailored views of conversation quality.

To build a custom report using Monitor metrics, go to Reports > + New report > Create your own and select the metrics you need from the Monitors category. You can filter by scorecard, time period, or any other attribute to focus on the segments most relevant to your team.

Scorecard criteria evaluation are qualitative data points used to categorize or filter your metrics.

To edit scorecards and monitors and score conversations, teammates need both of the following permissions:

Teammates who don't have both permissions can't see human QA Monitors and can only see reviews of their own work via the Reviews received view. They can't override AI-scored criteria on their own reviews.