Use Evals to build realistic test conversations, run them against your Fin configuration, and get automatic pass/fail results before any change reaches a real customer. Run the same tests again whenever you make a change, to check that Fin still behaves the way you expect.
Evals are built around two concepts:
Evals are themed groups of tests, like “Refund requests,” “Escalation scenarios,” or “Tone and friendliness.”
Simulations are the individual tests inside an Eval. Each one is a realistic, multi-turn conversation between a simulated customer and Fin, along with the checks Fin’s behavior is scored against.
When you run an Eval, every Simulation inside it runs automatically and gets a pass/fail result. An LLM (large language model) judge scores each Simulation against the checks you set.
Note: To create, edit, or run Evals, you need Can view configuration and propose changes in a draft release or Can view configuration and publish changes directly to main (individually or in a release) under Fin AI Agent & Automation. Teammates with Can view configuration can see Evals and their results.
Get started with Evals:
How to manage your Evals: find, filter, tag, and create Evals, and import Simulations from a CSV file.
Set up Simulations for Evals: choose the simulated customer, the data Fin sees, and the checks Fin is scored on, or create a Simulation from a real conversation.
How to run Evals and read the results: run Simulations, understand each result, and test in other languages.
How to add a Scorecard to an Eval: score the quality of Fin’s answers, as well as pass or fail.
Create and run Evals with Operator: let Operator create, run, analyze, and schedule Evals for you.
When should you use Evals?
Use an Eval whenever you want confidence in how Fin behaves:
Before shipping a change: to content, guidance, Procedures, or anything else in Fin’s configuration.
While working on a Release: to test the changes in your Release together, before they go live.
After a fix: to confirm you solved the problem you set out to solve.
On an ongoing basis: to re-run your most important Evals as a regression suite and catch unexpected changes before customers do.
Before going live with Fin: to check your configuration works as expected before customers talk to Fin for the first time.
Testing a Release with Evals
When you’re working on changes to Fin in a Release, run your Evals against those changes before they go live. Fin answers using the Release’s version of your content, guidance, Procedures, and Skills, so you can see how your changes perform without affecting any customers.
Open your Release and go to the Evals tab.
To create an Eval just for this Release, click Add an Eval, then New eval. It appears under Added in this release.
To check for regressions, click Run eval on any of your existing Evals. The tab shows how many of your Evals have been run against the Release (for example, 3/5 run).
Note:
New Evals and Simulations you create inside a Release don’t appear in your live Evals until the Release is merged.
Results are kept separately for each Release. Runs inside a Release don’t change the results you see in Fin Main.
If you change the Release after running an Eval, the results are marked out of date. Run the Eval again.
Inside a Release, you can’t rename or delete existing Evals, or edit a Scorecard.
Tags aren’t saved to a Release. Tag changes you make inside a Release apply to your live Evals straight away.
Running Evals isn’t required to merge a Release, but we strongly recommend it.
For more about Releases, see Releases for Fin.
Which testing tool should I use?
| Evals | Procedure Simulations | Batch Test |
Tests | Fin’s whole configuration, end to end | One Procedure on its own | Single questions |
Conversation type | Multi-turn | Multi-turn, within one Procedure | Single-turn |
Scoring | Automatic, by the LLM judge, plus optional Scorecards | Automatic, against success and outcome criteria | Manual (Good, Acceptable, or Poor) |
Best for | Regression testing and full customer journeys | Building and checking a Procedure | Quick informational checks |
Use Procedure Simulations while you’re building a Procedure, to check that it behaves correctly on its own. See Run Simulations for Fin Procedures.
Use Evals once that Procedure is live, to test the full conversation, including how it works with your content, guidance, data connectors, and handoffs.
If you use Batch Test today, start building equivalent scenarios as Evals. You can export your Batch Test questions to a CSV file and import them as Simulations.
Note: From November 5, 2026, new workspaces won’t see Batch Test in the left-hand navigation. If you already use Batch Test, it’s still available.
Simulation usage limits
There’s a monthly limit on the number of Simulations you can run in Evals. The limit applies to your whole workspace and resets on the first day of every calendar month. Your allowance is based on your workspace’s conversations in the last 30 days, and is re-checked each month.
Conversation volume | Simulations per month |
Under 1K | 250 |
1K–15K | 1,000 |
15K–100K | 1,750 |
100K–1M | 5,000 |
1M+ | 12,500 |
Each Simulation run counts as soon as it starts, including runs you stop partway through. If you reach your limit, you’ll see “You have reached your monthly simulation limit,” and you can’t run more Simulations until next month. If you click Run all close to your limit, the Simulations that fit within your allowance run, and the rest don’t start.
Current limitations
Up to 50 Simulations per Eval, and up to 50 rows per CSV import.
Tags apply to Evals only. You can’t tag individual Simulations, or run every Eval with a tag in one go.
No notifications. You won’t be notified when a run finishes.
No reviewer annotation yet. There’s no way to record whether a reviewer agrees or disagrees with a result.
No pre-built templates yet. Common test categories, like prompt injection, don’t have a ready-made starting point.
Workflows aren’t simulated. A Handoff check can confirm that Fin passes a conversation to a workflow, but Evals don’t simulate what the workflow does next.
FAQs
Do I need Operator to use Evals?
Do I need Operator to use Evals?
No, every Fin customer can create and run Evals from Fin AI Agent > Test > Evals. Operator is part of the Pro add-on and can create, run, analyze, and schedule Evals for you.
How long does an Eval take to run?
How long does an Eval take to run?
Expect minutes rather than seconds, because each Simulation is a full multi-turn conversation followed by scoring. Runs continue in the background, so you can leave the page and come back for the results.
Why did my Simulation fail when Fin’s answer looked right?
Why did my Simulation fail when Fin’s answer looked right?
If you didn’t set a Handoff check, the Simulation is scored against a default No handoff check, so it fails if Fin hands off. See How to run Evals and read the results.
Can I schedule Evals to run automatically?
Can I schedule Evals to run automatically?
Yes, with an Operator Routine. See Create and run Evals with Operator.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts