Use this article to choose the right way to test Fin: Evals for end-to-end conversations and regression suites, Procedure Simulations for a single Procedure, Batch tests for sets of single-turn questions, and Previews for quick manual checks while you build.

Quick reference: which tool does what

The table below compares the four ways to test Fin. Evals test Fin end to end, where Procedure Simulations test one Procedure at a time. See Evals for Fin.

	Evals	Procedure Simulations	Batch tests	Previews
What it is	Automated, end-to-end testing of Fin's whole configuration. Each Simulation in an Eval is a multi-turn conversation between a simulated customer and Fin, scored against checks you set.	Automated, AI-powered end-to-end testing of a single Procedure. AI acts as a simulated customer and judges the outcome against success criteria you define.	A question-and-answer testing tool for validating Fin's responses across up to 50 questions at once. Tests how Fin answers across content, guidance, users, languages, and automations.	A manual, interactive testing panel available throughout Fin's training areas. You type messages and see Fin's response and event log in real time.
Best for	Regression testing after any change to Fin. Testing full customer journeys, including what happens before a Procedure is triggered and after it hands off. Testing the changes in a Release before they go live.	Validating complex Procedure logic end-to-end before going live. Catching regressions after Procedure updates. Testing edge cases and branching paths.	Checking content coverage across many questions at once. Diagnosing gaps in knowledge, guidance, or automations. Pre-launch QA across audiences, brands, and languages.	Quick manual spot checks while building. Verifying how a specific change — to content, guidance, attributes, escalation, or a Procedure — affects Fin's response in the moment.
Works with	Everything Fin uses to answer: content, guidance, escalation guidance, Skills, and Procedures.	A single Procedure. For Simulations across Fin's whole configuration, use Eval Simulations. Fin for Sales has its own Simulations.	All customers using Fin — whether using Procedures, Tasks, or just knowledge content.	All of Fin's training areas: Content, Guidance, Attributes, Escalation, Procedures, and Deploy.
Testing mode	Automated — AI generates the customer side of the conversation and an LLM judge scores the result. Optional Scorecards also score answer quality.	Automated — AI generates the customer-side of the conversation and evaluates the result with a pass/fail judgment.	Automated — Fin generates a response to each question. You review and rate answers manually (Good / Acceptable / Poor).	Manual — you type messages yourself and observe Fin's response and event log.
Creates a real conversation?	No — Simulations run inside Evals and never reach a customer.	No — runs entirely in the background. No inbox conversation is created.	No — simulation only. Does not create inbox conversations or affect customer data.	Yes — Preview conversations appear in your inbox, but are excluded from reporting.
Hits live APIs?	No — data connectors always return their saved test response, never live data.	No — does not call live external APIs (e.g. Shopify, Stripe). Tests logic only.	No — data connectors use configured test/sample payloads. No live API calls.	Yes — can access real user data and live integrations when testing as a specific user or lead.
Saves & reruns	Yes — Evals are saved and can be rerun as a regression suite, or scheduled with an Operator Routine. Up to 50 Simulations per Eval.	Yes — simulations are saved and can be rerun whenever the Procedure is updated. Supports Run all.	Yes — test groups (up to 50 questions each) can be saved, rerun, and shared across teammates.	No — one-off manual conversations. Not saved or rerunnable.
Pass/fail judgment	Yes — the LLM judge gives each Simulation an automatic Pass or Fail.	Yes — AI evaluates the conversation against success criteria and assigns Pass or Fail with full reasoning.	No — you manually rate each answer as Good, Acceptable, or Poor and choose a root cause.	No — no automated judgment. You assess the result yourself.
Usage limits	Yes — monthly Simulation limit based on workspace conversation volume (250–12,500/month), reset on the first of each month.	Yes — monthly run limit based on workspace conversation volume (250–12,500/month). Included with Procedures at no extra charge.	No charge. Up to 50 questions per test group.	No limits.
Availability	Available to every Fin customer. No beta or access request needed.	Generally available to all customers using Procedures.	Available to existing Fin customers with a full seat and All conversations permission. From November 5th, 2026, new workspaces won’t see Batch tests and use Evals instead.	Generally available to all Fin customers.

When to use each tool

Use Evals when

You want to test a full, multi-turn customer conversation across everything Fin uses to answer: content, guidance, escalation guidance, skills, and Procedures.
You want automatic pass/fail results from checks you define (Fin reply, Procedure triggered, data connector called, handoff), plus optional quality scoring with a Scorecard.
You want a regression suite you can re-run after any change to Fin, or schedule with an Operator Routine.
You want to test the changes in a Release before they go live.
You want to recreate a real conversation where Fin's answer wasn't good enough and keep re-testing it until it's fixed.

Learn how to set up Evals in Evals for Fin.

Use Procedure Simulations when

You need to validate a Procedure end-to-end before setting it live.
You want automated, repeatable tests that catch regressions after every update.
Your Procedure has branching logic, Conditions, or sub-procedures — and you need a separate test case per path.
You want AI to judge the outcome against defined success criteria (Fin reply, attribute values, data connector calls, instruction outcome).
You need to test time-sensitive logic (e.g. refund windows) by controlling the simulation time and data values.

Note: Simulation time only controls the date Fin's own reasoning uses for plain-language date logic. It does not change what a Python code condition sees — code that calls datetime.datetime.now() always reads the real system date, regardless of the Simulation time you set. To test a code-based date branch, run the simulation on the actual calendar date, or rewrite the condition in plain language instead of code.

Use Batch tests when

You want to validate Fin's answers across a broad set of questions — not just one scenario.
You're doing pre-launch QA across content, guidance, audiences, languages, or brands.
You want to diagnose content gaps by seeing which sources Fin pulls from for each answer.
You want to test questions generated from real past conversations or AI topics.
You need a downloadable CSV report to share results with your team or senior leaders.

Use Previews when

You're actively building and want to spot-check a change immediately — to content, guidance, an attribute, an escalation rule, or a Procedure.
You want to manually walk through a conversation as a specific user, audience, or in a specific language.
You need to see Fin's event log to understand exactly what it applied (personality, guidance, tasks, content sources).
You want to test Fin Vision by sending an image or screenshot.
You're validating a Procedure flow and want to see the full customer and teammate inbox experience (including downstream automations).

FAQs

What's the difference between Simulations and Previews?

Previews are manual and interactive: you type messages yourself and watch Fin respond in real time. They're great for quick spot checks while building.
Simulations are automated: AI acts as the customer, runs the full conversation, and judges the outcome against success criteria you set.
Simulations don't create inbox conversations; Preview does. Use Simulations when you need to validate Procedure logic at scale. Use Preview for in-context manual testing.

What's the difference between Simulations and Batch tests?

Batch tests check how Fin answers a broad set of questions — testing content coverage, guidance, and automations across many scenarios at once. You rate answers manually.
Simulations test a single Procedure end-to-end with automated AI judgment.
Batch tests work with all Fin setups. Procedure Simulations are exclusive to Procedures.

What's the difference between Batch tests and Previews?

Batch tests are structured, repeatable QA runs across up to 50 questions — useful for pre-launch checks and diagnosing content gaps.
Previews are unstructured and manual — one conversation at a time, useful for instant feedback while building.
Batch test results are saved and can be exported to CSV. Preview conversations appear in your inbox but aren't saved as test runs.

Do Simulations call live APIs or affect real data?

No. Simulations do not call live external APIs (like Shopify or Stripe) and do not affect any real data. They test logic only. If you need to test with real API calls, use Preview and select a specific user with real data in their external system.

Will I be charged for Simulations or Batch tests?

Simulations are included with Procedures at no extra charge. Batch tests are also free to use (you won't be charged for answers generated through Batch test).

Note: Simulations do have a monthly run limit (250–12,500 per month depending on conversation volume), which resets on the first of each month.

Can I use Batch tests to test a Procedure?

Batch tests show whether a Procedure would have been triggered for a given question, but they don't execute the Procedure or evaluate its end-to-end logic. For full Procedure validation (including branching, data connector calls, and instruction outcomes) use Simulations.

Which testing tool should I use before going live?

Use the tools in combination before going live: Preview for quick manual checks as you build, Procedure Simulations to validate each Procedure's logic with automated pass/fail results, and Evals to confirm Fin's full configuration handles your most important customer conversations end to end. Batch tests remain available for single-turn question checks.

Running Simulations before every Procedure launch is strongly recommended.

What's the difference between Evals and Simulations?

Simulations test one Procedure. Each Procedure Simulation runs within the bounds of that Procedure, so it's the right tool while you build a Procedure and for checking it still works after changes.
Evals test Fin end to end. An Eval Simulation runs a full conversation across everything Fin uses to answer, including what happens before a Procedure is triggered and after it hands off. Evals also replay real conversations and run as a regression suite after any change to Fin.
Use both: Procedure Simulations while building a Procedure, then an Eval to cover the full customer journey once it's live. See Evals for Fin.

Should I use Evals or Batch tests?

Evals do everything Batch tests do, and more. Batch tests check single-turn questions that you rate by hand. Evals run full multi-turn conversations, score them automatically against checks you define, and can be re-run as a regression suite. If you use Batch tests today, import your questions into an Eval with CSV import, or ask Operator to build the Eval for you.

Key facts to remember

Procedure Simulations are exclusive to Procedures. They cannot be used with Tasks, Workflows, or general content testing. To simulate full conversations across Fin's whole configuration, use Eval Simulations. Fin for Sales has its own Simulations.
Batch tests are available to existing Fin customers — whether using Procedures, Tasks, or just knowledge content.
Preview conversations appear in the inbox but are excluded from reporting.
Only Preview can hit live APIs. Simulations and Batch tests are safe, sandboxed environments.
Simulation run limits are per workspace, not per teammate, and reset monthly.
Answer ratings in Batch test do not train Fin — they're for human review and root cause analysis only.
In Simulations, if a test passes but the success criteria are too rigid (e.g. requiring a specific question Fin found itself), the test may still fail. Focus success criteria on outcomes, not intermediate steps.
Evals test Fin's whole configuration in one multi-turn conversation, including Procedures, content, guidance, escalation guidance, and skills. Use them as your regression suite after any change to Fin.

💡Tip

Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts

Evals vs. Simulations vs. Batch tests vs. Previews