Use this article to set up a Simulation: define the simulated customer, control the data Fin sees at the start of the conversation, choose the checks and handoff the LLM (large language model) judge uses to score Fin, or create a Simulation from a real conversation.
How to open the Simulation editor
To set up a Simulation, open it and click Edit, or click New simulation on an Eval page. The editor has three sections: the simulated customer, the data Fin sees, and how the LLM judge scores Fin. When you finish, click Save, then click Run to test it.
How to define the simulated customer
The Define simulated customer section sets up who Fin will be talking to.
Title (optional): if you don't add one, the Simulation name matches the customer's first message.
Simulating as: the customer Fin talks to, for example Test customer, and the brand, shown as All brands.
Channel: Chat or Email.
Simulation time: a fixed date and time for the Simulation to use. It uses the customer's time zone, or GMT (Greenwich Mean Time) if none is set. It starts as Not set.
Customer's first message (required): the first thing the simulated customer says to Fin.
Follow-up instructions: what the customer knows, what they'll share if asked, and how they communicate. Fin uses them to generate the rest of the customer's side of the conversation, for a realistic multi-turn simulation.
How to choose who the simulated customer is
By default, every Simulation runs as a Test customer. The Test customer belongs to all of your audiences, so Fin can use all of your live content, guidance, and automations. That's useful for checking what Fin knows, but it doesn't show what a specific customer would experience.
To test audience targeting, simulate a real customer or lead from your workspace instead. Fin then sees only the content, guidance, Procedures, and data connectors that customer qualifies for, just as it would in a live conversation.
In the Simulation editor, go to Define simulated customer.
Next to Simulating as, click Test customer and select Customer or lead.
To narrow the list, choose an audience from the dropdown under the search box. It shows Everyone by default, and lists the users and leads currently in that audience.
Search by name, email, or customer ID, and select the person you want Fin to talk to.
When the Simulation runs, it uses that person's attributes, company, and tags. To test a specific scenario, override any of these in Define Fin data (optional) > Starting attribute values. Anything you don't set uses the customer's existing values.
Note:
Audiences that use conversation attributes, or that filter on brand, can't be selected, because membership depends on a conversation that hasn't happened yet. You can still pick a customer using another audience, or by searching everyone.
The audience is only used to find a customer and isn't saved with the Simulation. Each run checks which audiences the customer belongs to using their data at that moment, so results can change.
Only users and leads can be selected. Visitors aren't available.
If you switch back to Test customer (default), audience targeting is turned off for that Simulation.
How long can a Simulation run, and how many can an Eval hold?
A Simulation ends after up to 6 replies from Fin, or earlier if Fin resolves the conversation, hands off, or the simulated customer ends the conversation. Each Eval supports up to 50 Simulations. To add more, delete existing Simulations or put the extra Simulations in a new Eval.
How to define the data Fin sees in a Simulation
The optional Define Fin data section controls the data Fin sees at the start of the Simulation.
Starting attribute values: the attribute values Fin sees at the start of the conversation. Click Add attribute to set one.
Data responses: in Simulations, data connectors always return their saved test response, never live data, regardless of who you're simulating as. Click Customize a connector response to change the test response for this Simulation only.
How to define how the LLM judge scores Fin
The Define LLM judge's behavior section defines how Fin's answers are scored. It has two parts: Simulation checks and Handoff.
How to add a Simulation check
Simulation checks describe what Fin should say or do to pass, if anything. To add one, click Add and choose a check type:
Fin reply: describe what Fin should say during the conversation. Separate each expected response into its own success criterion.
Procedure triggered: Fin started a specific Procedure.
Procedure switched: Fin moved from one Procedure to another during the conversation.
Skill: Fin used a specific Skill.
Data connector: Fin called a specific data connector.
How to set the expected handoff
Handoff describes what should be true by the end of the conversation. Choose one of these options:
No handoff: Fin didn't hand off to a team or workflow.
Handed off to team or teammate: Fin couldn't answer, or handoff guidance has been applied.
Handed off to a workflow: the conversation isn't simulated after the handoff.
Important: If a Simulation has no Handoff check and no Simulation checks, it's scored against a default No handoff check, and it fails if Fin hands off. This default isn't shown in the editor, so add an explicit Handoff check to each Simulation.
After you set up a Simulation, see How to run Evals and read the results.
How to create a Simulation from a real conversation
To create a Simulation from a real customer conversation, add one of Fin's answers to an Eval from the Inbox, or ask Operator to create the Simulation. Either way, you don't have to type the scenario yourself.
A Simulation created from a real conversation is called a replay. A replay captures the conversation up to the Fin answer you chose and freezes the earlier messages in place. Each time it runs, Fin generates only that one reply again, so you're testing the response you care about, not a conversation that changes every run. This makes replays the fastest way to turn a real-world miss into a test you can keep.
How to add a Fin answer to an Eval from the Inbox
To add a Fin answer to an Eval from the Inbox, open the conversation, choose the Fin message, and add it to an Eval.
Open the conversation in the Inbox.
Hover over the Fin message you want to test, then click the options menu (...) that appears next to it.
Click Add Fin answer to Eval.
In the Add to eval dialog, click Select an eval and choose the Eval to add it to.
Click Add.
Add Fin answer to Eval only appears on Fin answers that can be replayed.
How to create a Simulation with Operator
To create a Simulation with Operator, ask it to create a simulation for the conversation. Operator replays the customer's opening message and proposes the Simulation in the Eval you choose.
Note: To create Simulations with Operator, you need the Pro add-on.
In a chat with Operator, write Create a simulation for this conversation and add the conversation.
Answer the two questions Operator asks: Which eval should the simulation go in? and How should Fin end the conversation? For example, choose Fin answers it itself (no handoff) if Fin should resolve the conversation without handing off.
Review the proposed Simulation, then click Publish to main to add it to the Eval.
Until you click Publish to main, the Simulation stays a pending change and can't run.
How to edit a replay Simulation with Operator
To edit a replay Simulation, ask Operator to make the change in the same chat, for example update it to handoff to a teammate. Operator proposes the edit to the Simulation checks, and you click Publish to main to save it.
What does a replay Simulation include?
After you click Publish to main, the replay Simulation appears in the Eval you chose, named after the customer's opening message. Open it from the Eval to see these parts:
Initial conversation context: the source conversation's messages up to the customer's message, with a placeholder where Fin's single reply appears when the Simulation runs.
View source conversation: a link to the original conversation.
Define simulated customer: Simulating as, the brand, the Channel, and the Simulation time.
Follow-up instructions: optional instructions for what the customer says after Fin's first reply. Adding them requires at least one Simulation check, so there's something for Fin to be scored against.
Note: Some parts of a replay Simulation can't be edited, including the Define simulated customer section. They come from the source conversation. To change what Fin is checked on, edit the Simulation checks, for example by asking Operator to update them.
How to use a replay to fix Fin's answer
To fix a Fin answer with a replay, run it, change Fin's configuration, and re-run it until it passes:
Run the replay to see how Fin answers with your current configuration.
Make a change to the content, guidance, Procedure, or Skill behind the answer. To test the change before it goes live, make it in a Release.
Run the replay again and compare the new answer with earlier runs, using the run history dropdown on the Simulation.
Add checks that define what "fixed" means, such as a Fin reply check or a Handoff check, so you get a clear pass or fail instead of a judgment call.
Keep the replay once it passes. It now works as a regression test, so re-run it after future changes to make sure the fix holds.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts