How to run Evals and read the results

Use this article to run the Simulations in an Eval, read the results on an Eval page, and review how Fin did on a single Simulation.

To run every Simulation in an Eval at once, open the Eval and click Run all. To run one Simulation, click the run button on its row. While a Simulation runs, its Last run column shows Just now and its Result column shows Running. When it finishes, Result changes to Pass, Fail, or Error.

When you run an Eval, results aren't instant, because each Simulation is a full multi-turn conversation. Fin replies, the simulated customer responds, and the conversation continues until it ends. After that, the checks and any Scorecard are scored. A Simulation can take a few minutes, and a full Eval can take longer.

To run only some Simulations, select them and click Run, or search or filter the list and click Run filtered.

Your workspace can run a set number of Simulations in Evals each month. The limit applies to the whole workspace and resets on the first day of every calendar month.

Your monthly Simulation allowance is based on your workspace's conversation volume in Intercom, and larger workspaces receive higher allowances.

The table below shows the monthly Simulation allowance for each conversation volume segment.

When your workspace reaches 80% of its monthly Simulation limit, a yellow banner shows your current usage (for example, 850/1000) and when the limit resets.

At 100%, a red message appears, and you can't run more Simulations until the next month.

An Eval page shows the results of one Eval and lists the Simulations inside it. To open it, click an Eval's name on the Evals page. The Eval's name appears in the page title, after the Evals breadcrumb.

The Simulation results count covers the number of Simulations in the Eval that passed all Simulation checks and Handoff criteria when they were last run. To see this definition, hover over the question mark next to Simulation results.

Click the ⋯ menu at the top of an Eval page to choose one of these options: Edit eval, Import CSV, or Delete eval.

A Simulation page shows one simulated conversation, the checks it was scored on, and how Fin did. To open it, click a Simulation's row on the Eval page. The page title shows the path Evals, then the Eval's name, then Simulation.

To change what a Simulation checks, see Set up Simulations for Evals.

When a Simulation finishes, it shows Pass, Fail, or Error. A Simulation passes only if every Simulation check and Handoff check passes, so a single failed check means the Simulation fails. For each Simulation you get the overall result, the full conversation transcript, and the event log showing Fin's thinking and the tools and information it used at every turn.

If a single conversation needs Fin to work across multiple Procedures, you see Fin switch between Procedures as the simulated conversation unfolds, just as it would for a real customer. Each Simulation keeps its last 15 runs, and you can use the run history dropdown on a Simulation to look back at earlier results.

Each Eval on the Evals list shows one overall result. These are also the options in the Result filter:

Passed, Partially passed, and Failed Evals show a count, such as 8/10 Passed. The badge is green when every Simulation passed, and red when fewer than half did.

A Simulation can fail on the default No handoff check even when Fin's answer looked right. If you add a Simulation with no explicit Handoff check and no Simulation checks, the Simulation passes only if Fin resolved the conversation without handing off to a team or workflow. This default isn't shown in the Simulation setup, so check whether a Handoff check was set when a Simulation fails and you can't see why.

To test Fin in another language, write the customer's first message in that language. Simulations support the same languages as Fin, and you don't need to set a language. When a Simulation starts, Fin detects the language of the first message using the same language detection as live conversations, and uses it for the whole Simulation. Fin replies in that language, the simulated customer writes its follow-up messages in it too, and Scorecards are scored in it.

To test the same scenario in several languages, create one Simulation per language, and use tags on the Evals to group them. Write opening messages of at least a full sentence so language detection is reliable. The language is set once at the start and doesn't change if the customer switches language mid-conversation.

For the full list of supported languages, see Fin's supported languages.

After you change Fin's content, guidance, Procedures, or Skills, re-run your existing Evals to check for regressions. Every Simulation is stored and reusable, so re-running takes one click rather than recreating your test scenarios. To run Evals on a regular schedule, such as every Monday morning, set up a Routine in Operator (see Create and run Evals with Operator) that runs your chosen Evals at the times you pick. Scheduling isn't available on the Evals page.