Skip to main content

Create and run Evals with Operator

Ask Operator to create, run, and analyze Evals for you, then review its changes and save them to main or a release draft.

Written by Dawn Perrott

Use this article to ask Operator to create, run, and analyze Evals, review the changes it proposes, and schedule recurring runs. Operator can also recommend improvements to your Evals and Simulations or to Fin's configuration. For the basics, including what Evals and Simulations are, see Evals for Fin.

Note: To use Evals with Operator, your workspace needs the Pro add-on and you need access to Operator.

How to create an Eval with Operator

To create an Eval with Operator, go to Fin AI Agent > Operator and ask it to create an Eval for your use case.

The Operator new chat screen. The sidebar lists New chat, Routines, Artifacts, and Customize. The message box holds a prompt that reads: Create a new Eval called Refund requests with 5 Simulations, covering a straightforward refund request, a refund outside the refund window, a customer charged twice, a customer asking in Spanish, and a handoff to the Billing team.

For example, you could ask it to test:

  • The Release you're working on: the changes to Fin's configuration in your current Release, before they go live.

  • Specific conversations: share one or more conversation IDs, and Operator builds Simulations that recreate them.

  • Edge cases you haven't covered yet: ask Operator to look for gaps in your existing Simulations and suggest new ones.

  • A new or updated Procedure: check that Fin follows each step, including the points where it should hand off.

  • A topic: name a topic, such as refunds, and Operator creates Simulations covering the range of scenarios customers are likely to contact you about.

  • More variations of an existing Eval: ask Operator to add Simulations with different customers, tones, or languages.

You can tell Operator what the Simulations, Simulation checks, and Handoff checks should be, or let it work them out. If anything is unclear, such as whether to add Simulations to an existing Eval or which team a handoff should go to, Operator asks you first. Choose an option, type your own answer under Something else, or click Skip to let Operator decide.

Tip: The more specific your request, the fewer questions Operator needs to ask.

Example prompt: Create a new Eval called "Refund requests" with 5 Simulations: a straightforward refund request, a customer asking about a refund outside the refund window, a frustrated customer who's been charged twice, a customer asking in Spanish, and one where Fin should hand off to the Billing team. To adapt it, replace the Eval name and the scenarios with your own topic and cases.

How to review and save what Operator creates

Operator doesn't save anything until you review it. New Evals and Simulations appear in the "changes pending" panel above the message box, each marked "New". Click any item to review it before saving.

An Operator chat named Eval creation setup, with a new Simulation open in a panel on the right. Above the message box, a 5 changes pending panel lists five new Simulations, each marked New, with a Publish to main button. The Simulation panel shows Setup, Configuration, Customer's first message, Follow-up instructions, and Simulation checks.

When you're ready, choose where to save the changes:

  • Publish to main: saves all pending changes to your live Evals straight away.

  • Save to release draft: saves the changes to an existing release draft, so you can test them alongside other changes before they go live. Select a draft from the list, or click New release draft to create one.

    The Publish to main menu open on a new Simulation. The first option, Publish to main, says changes will be saved to main and available for Fin to use. Below it, a Save to release draft section lists existing release drafts, followed by a New release draft option.

To discard the changes instead, click the delete icon.

The header of a new Simulation panel, with the pointer over the delete icon next to the Publish to main button. A tooltip reads Discard change.

How to run an Eval with Operator

To run an Eval with Operator, ask it to run the Eval, for example: Run the Refund requests Eval. Operator shows you the results as each Simulation runs.

An Operator chat after the changes were applied. The chat shows You applied 5 changes and a Run them message, followed by a Running 5 simulations card that lists the opening message of each Simulation.

You don't need Operator to run an Eval. Go to Fin AI Agent > Test > Evals and run it yourself. Runs started from Operator appear there too, so you can review every result in one place.

How to read the results Operator gives you

Once every Simulation has run, Operator lays out the results and explains them:

  • A results table: each Simulation's overall result (Passed, Failed, or Error) and the individual checks behind it, so you can see why it landed that way. For example, a Simulation called "Bulk workspace export" can fail because the Handoff check failed (Fin escalated when it shouldn't have), even though the Fin reply check passed.

  • A summary: the headline counts (3 passed, 1 failed, 2 errored), followed by a plain-language breakdown of what went wrong and why. For example, Operator can trace a failed export back to a guidance rule that hands off to a teammate whenever the word "failure" appears.

How to fix issues with Operator

After analyzing the results, Operator can recommend improvements, either to your Evals and Simulations or to Fin's configuration, and make those changes for you. For example, when a Simulation fails because of a guidance rule, Operator traces the failure back to that rule so you can see what to fix. Operator doesn't apply fixes on its own. Each change it proposes appears in the "changes pending" panel for you to review before it's saved, as described in How to review and save what Operator creates.

How to schedule Evals with Operator and Routines

To run Evals on a regular schedule, such as every Monday morning, set up a Routine in Operator. You can’t schedule Evals from the Evals page.

To schedule an Eval:

  1. Go to Fin AI Agent > Operator and click Routines.

  2. Under Set up a routine, describe what you want to run and when. For example: “Run the Refund requests Eval every Monday at 9am and summarize the results.”

    The Routines page in Operator. A Set up a routine box holds the request: Can you run Eval Refund requests every Monday morning? Below it are a list of existing Routines and a list of suggested Routines.

  3. Operator drafts the Routine and shows it in a panel on the right, with its trigger, schedule, and instructions. If anything is missing, such as the time or time zone, Operator suggests a default and asks you to confirm or change it.

    A draft Routine called Weekly run of Refund requests eval, shown in a panel next to the Operator chat with Accept and Discard buttons. The panel lists the trigger as Scheduled, the schedule as every Monday at 9:00 AM, and the instructions. In the chat, Operator asks whether 9:00 Monday works, with the options 9:00 Monday is fine, Earlier, and Later.

  4. Check the Routine’s details, then click Accept to turn it on, or Discard to cancel it.

    The Routine Weekly run of Refund requests eval after it was accepted, marked Active, with Run now, Edit, and options buttons. The Details tab shows the trigger as Scheduled, the schedule as every Monday at 9:00 AM, and the instructions.

Each time the Routine runs, it reports how many Simulations passed and failed, which ones failed and why, and any recommended fixes to Fin's content, guidance, or Procedures.

💡Tip

Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts

Did this answer your question?

Related Articles