Writing effective criteria is what separates a Monitor that surfaces real issues from one that floods your queue with noise. This guide covers best practices for both Monitor flag criteria and Scorecard attribute descriptions. Monitors currently evaluate Fin AI Agent conversations only.
Note: Monitors is available as part of the Pro add-on.
Monitor flag criteria vs. scorecard attribute descriptions
These two types of criteria work differently, so they need to be written differently.
| Monitor flag criteria | Scorecard criteria descriptions |
Purpose | Decides which conversations get reviewed | Defines how each conversation is evaluated |
Logic | Yes/no - each monitor runs independently | Competitive - AI selects the single best match |
Key challenge | Reduce false positives and false negatives | Eliminate overlap between criteria values |
Best practices for writing Monitor flag criteria
Monitors run as independent yes/no checks. Multiple Monitors can flag the same conversation - and that is fine. Because of this, precision matters more than distinction.
1. Describe observable behavior, not inferred intent
Focus on what appears in the conversation.
Avoid: Customer is frustrated
Prefer: Customer uses phrases such as This is unacceptable, I am extremely disappointed, or This is ridiculous.
The AI performs better when evaluating explicit signals rather than emotional interpretations.
2. Include concrete examples
Examples dramatically reduce ambiguity.
Use explicit phrasing patterns: e.g., cancel my subscription, close my account, delete my data
Examples anchor the model to real-world language.
3. Add explicit exclusions
Reducing false positives is critical for Monitors.
Example: Customer uses profanity. EXCLUDE: mild language such as damn or crap. If something should not trigger the monitor, say so clearly.
4. Use quantifiable thresholds
Avoid vague wording.
Bad: Fin gives a short response.
Better: Fin response is fewer than 50 words.
Specific thresholds improve consistency.
5. Break multi-step logic into numbered criteria
If your Monitor depends on sequence or pattern, structure it clearly:
Customer expresses frustration.
Fin responds without acknowledging emotion.
Customer repeats complaint.
This makes the logic deterministic and easier to evaluate.
6. Keep it simple
If the rule is straightforward, do not overcomplicate it.
Example: Fin suggests next steps (e.g., Please try clearing your cache, Log out and back in, Click this link).
Clarity beats complexity.
7. Use 'explicitly' to require direct customer language
If your Monitor should only trigger when a customer directly states something — not just implies it — include the word "explicitly" in your criteria. Without it, the AI may infer intent from context and match conversations where the behavior was only suggested, not stated.
Without "explicitly": Customer requests a call back — could match "Can you connect me to the security team?" since the AI may infer this implies a request for direct contact.
With "explicitly": Customer explicitly requests a call back — only matches if the customer directly asks, e.g., "Can I get a call?" or "Please call me."
Tip: Use the Test Monitor tool to validate your criteria against real conversations before turning it on. Update the flag criteria and rerun the test until results accurately reflect what you want the Monitor to capture.
8. Use precise filters for assignment conditions, not flag criteria
Flag criteria reads the conversation transcript — it doesn't have reliable access to a structured assignment history. Avoid writing conditions like "only if Agent X was the assigned teammate when the conversation closed" or "don't flag if it was reassigned before closing" in flag criteria. No matter how carefully worded, these conditions introduce false positives because the AI is interpreting unstructured text, not querying an assignment record.
Instead, use the conversation filters available under Add filters to scope the monitor by assignee directly — these check the conversation's actual assignment record, not the transcript, and apply deterministically with no AI interpretation involved.
Tip: If you find yourself writing exclusion sentences to patch around an assignment condition in flag criteria, that's a signal the condition belongs in a precise filter instead.
Best practices for writing scorecard criteria descriptions
Start with the core principle: criteria compete. The AI looks at the full list and selects the single best match for each criteria. Your job is to make that choice obvious.
1. Use clear, concise names
Keep names short and specific. Someone reading the list should immediately understand the purpose without opening the description.
Bad: Customer Communication Issues
Better: Tone - Rude or Dismissive
2. Write comprehensive descriptions
Descriptions carry most of the classification signal.
Explicitly describe all conversation types that belong.
Include keywords, common phrasings, and examples.
Think through edge cases and include them.
Clarify what good and bad instances look like.
The description should make it easy for the AI to recognise real-world phrasing, not just abstract definitions.
Give every rating option the same level of detail. If the Fail option lists ten phrasings and the Pass option is one line, the AI will find the Fail option easier to match and will over-select it.
3. Make criteria clearly distinct
Criteria within the same scorecard should not compete conceptually.
Avoid semantic overlap.
Ensure each attribute has a clear boundary.
If two attributes could reasonably apply for the same reason, refine one of them.
It is fine if a single conversation fits multiple criteria across the scorecard. What matters is that within each criteria set, the values are clearly separable.
4. Evaluate quality systematically
When reviewing your taxonomy, assess each criteria on:
Clarity / conciseness
Description comprehensiveness
Criteria distinction
Overlapping criteria (if any)
Final score + commentary
This structured review forces you to tighten definitions and reduce ambiguity - which directly improves classification performance.
5. Describe every rating option, including Pass and Not Applicable
The AI compares the conversation with each rating option's description and selects the best match. It is matching, not following steps, so the option with the richest, most recognizable description tends to win.
Write each option as a positive description of the conversations that belong to it, with keywords and one short example.
Describe the common case in the Pass option. If most conversations should pass, make Pass the easiest option to match.
Give Not Applicable its own concrete description (for example "no teammate replied" or "spam or test message"). An option that only says "N/A" is rarely selected.
Keep each option to three to five lines. Long exclusion lists inside the Fail option are not reliably applied; turn each exclusion into a line in Pass or Not Applicable instead.
Balancing the options reduces over-selection. It does not make the AI apply a rule it cannot apply; see points 6 and 7.
Bad: Fail: the customer says the answer is wrong, didn't work, doesn't match what they see, contradicts an earlier answer, asks "are you sure" (and ten more lines). Pass: anything else.
Better: Fail: after Fin answered, the customer explicitly said the answer was wrong or didn't work ("that's not right", "I already tried that", "that option isn't there"). Pass: the customer thanked Fin, asked a follow-up question, gave the details Fin asked for, or asked for a person without saying the answer was wrong. Not Applicable: Fin gave no answer to the question, or the message is spam or a test.
6. Avoid conditions that depend on the order of events
The AI does not reliably apply order in scorecard rating options. A condition such as "the customer asked for a person before Fin answered" or "accepted only after a teammate replied" is scored as its simpler form, "the customer asked for a person" or "a teammate replied", even when the Pass option describes the other order and gives an example.
If the behavior you care about depends on order, expect the AI to score the simpler version, and check a sample by hand before you report the number.
Where you can, write the criterion as the simpler, order-free behavior and separate the cases afterwards using conversation attributes or filters in reporting.
If the order is the whole point, score that criterion manually on a sample rather than with AI.
7. Keep judgement calls out of AI-scored criteria
The AI scores what it can see. A criterion that depends on a judgement, such as "the teammate added nothing material" or "the customer accepted the explanation itself rather than the handoff", will drift. Rewriting it around something the customer says helps only if the new wording does not also depend on the order of events. If it does, score it manually on a sample.
8. Test a changed criterion on a copy before saving it
Saving a change to a scorecard criterion and choosing "Going forward and backfill" re-scores every conversation the Monitor has reviewed, usually within a few hours. "Going forward only" leaves past reviews as they are. Either way, counts before and after the change are not comparable.
Copy the scorecard, make the change in the copy, and attach it to a second Monitor on a small random sample of the same conversations.
Export the per-criterion results after a few days and read 20 Fails and 10 Passes per criterion.
Only when the copy scores the way you expect, make the same change in the live scorecard, and note the date in the Monitor's description so anyone reading a trend knows where the break is.
Note: Testing wording with a separate AI assistant (for example, asking Operator to score conversations against the text) is a useful first pass, but it is not the same model as the Monitor's evaluator. Treat it as a draft check, not a sign-off.
Tip: If you want rating reasons to appear in the dataset export, add predefined reasons to the criterion. Without them, the rating reason column is empty.
FAQs
How long should my flag criteria be?
How long should my flag criteria be?
There is no fixed length - the right length is whatever it takes to describe the behavior precisely. A simple Monitor might only need two or three sentences. A complex one (like detecting multi-step failure patterns) may need a structured, numbered description. Err on the side of more detail rather than less. For scorecard rating options, add that detail evenly across Pass, Fail and Not Applicable (see "Describe every rating option" above).
Can I use the same scorecard criteria across multiple scorecards?
Can I use the same scorecard criteria across multiple scorecards?
Yes - criteria titles and descriptions are reusable. Once you have created criteria, you can add it to multiple scorecards. Note that previous rating scores cannot be reused and will need to be set from scratch in each scorecard.
What is the difference between monitor flag criteria and a scorecard criteria description?
What is the difference between monitor flag criteria and a scorecard criteria description?
Monitor flag criteria determines whether a conversation gets pulled into a Monitor at all - it is a yes/no filter. Scorecard criteria descriptions define how each conversation is scored once it is in the Monitor. Think of the Monitor as the net and the scorecard as the ruler.
My Fail counts jumped after I tightened a criterion. Why?
Two common causes. The Fail option became much more detailed than Pass, so it matches more conversations, not fewer: add equivalent detail to Pass and Not Applicable. Or the tightened wording depends on the order of events or on a judgement, which the AI scores as the simpler behavior: see points 6 and 7. In both cases, re-test on a copy before saving to the live scorecard.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts