Use this article to update your content with Operator, save the changes to a release instead of publishing them straight away, test how Fin answers with those changes using Evals, and then publish every change together. This process is useful when a change affects several articles at once, such as a pricing or policy update with a fixed go-live date.
Note: Operator is included as part of the Pro add-on. You'll need the Pro add-on to use Operator.
What does the release testing process look like?
The release testing process lets you prepare content changes, test them with Fin, and publish them together, without changing what customers see on Fin Main (the live version of your content that customers, Fin, and Copilot use) until you're ready. A release is a saved set of content changes that you test and publish together.
The process follows 5 steps:
Update your content with Operator, or edit it yourself in Fin Main.
Save the changes to a release draft instead of publishing them.
Create an Eval with Simulations that cover the questions your customers will ask.
Run the Eval on the release, and on Fin Main to compare.
Publish the release when the results look right.
For example, say your refund window is changing from 30 days to 14 days on a set date. You can update every article that mentions the refund window, save the changes to one release, check that Fin gives the new answer, and publish them all on the day the policy changes.
How to update content with Operator
Operator can find every article affected by a change and propose updates for you to review. To update content with Operator:
Open Operator.
Describe the change and ask Operator to find and update the affected content. For example: "Our refund window is changing from 30 days to 14 days. Find every article and snippet that mentions the refund window and propose updates."
Review each proposal in the review panel. You can see what's being added, removed, or changed before anything is saved.
Ask Operator to adjust any proposal that isn't quite right.
Tip: You can also edit an article yourself in Fin Main and save it to a release the same way. Learn more in how to publish, save, and unpublish articles in Fin Main.
How to save content changes to a release
Saving content changes to a release keeps them out of Fin Main, so customers and Fin keep using the current content until you publish the release. To save Operator's proposed changes to a release:
In the review panel, click the arrow next to Publish to main.
Choose an existing release, or click New release draft.
For a new release, enter a name and an optional description, then click Create release.
Repeat for each proposal you want in the same release.
After you save a change, a release icon with a count appears next to the Operator message box. Click the icon to see the release name, its status, and the items in it, then click Open release to go to the release.
You can also click the In release label next to a saved change in the chat to open the release.
Note:
Changes saved to a release aren't live. Customers, Fin, and Copilot keep using the Fin Main version until you publish the release.
If you edit an item directly inside a release, the editor doesn't save automatically. Click Save to release often, or you'll lose your edits if you refresh the page.
Tip: Learn more in how to use Operator with releases.
How to create and run an Eval for a release
An Eval is a set of saved test conversations, called Simulations, that Fin answers so you can check its responses. When you run an Eval inside a release, Fin answers using the release's version of your content, so you can see how your changes perform without affecting any customers. Each Eval can hold up to 50 Simulations.
How to create an Eval for a release with Operator
Ask Operator to create an Eval for your change. For example: "Create an Eval called Refund window update with Simulations for a customer asking how long they have to request a refund, a customer asking for a refund after 20 days, and a customer asking whether the policy has changed. Save it to the release draft Refund window update."
Review the Simulations Operator suggests in the changes pending panel above the message box. Each Simulation is one test conversation, such as a customer asking about the new refund window. Ask Operator to add, remove, or change any Simulation before you save.
Click Save to release. All the Simulations are saved to the Eval in your release.
Run the Eval from the release's Evals tab, click Run eval on the Eval's row. Every Simulation in the Eval runs using the release's version of your content.
Open the results to check each Simulation's answer and the content Fin used to answer it.
If a Simulation fails, update the content in the release, then run the Eval again. To edit an item in a release, open the release, click the item, click Edit content, make your changes, and click Save to release.
Tip: Use Preview in the release to chat with Fin as a customer would. Preview conversations use the release's changes, don't touch Fin Main, and aren't charged.
How to publish a release
Publishing a release makes every change in it live on Fin Main at the same time. To publish a release:
Open the release and check that your Evals have passed.
Click Release at the top right of the release page.
You can also click Release under the release name in the left sidebar.
Select Publish to Fin Main. To release the changes to a share of customers before everyone, select Roll out gradually instead.
After you publish, the release shows the status Published, and customers, Fin, and Copilot use the updated content straight away. You can also ask Operator to publish the release. Operator summarizes what will change and asks you to confirm before it publishes.
Important: A published release can't be undone as a whole. To reverse changes after publishing, use Roll back changes in the release. Learn more in how to roll out a release in Fin.
What are the best practices for testing content in a release?
These best practices help you get reliable results when you test content changes in a release:
Keep one change per release. Group the content for one change, such as a policy update, in its own release so you can test and publish it on its own.
Run a baseline first. Run your Eval on Fin Main before you make changes, so you can see what the release improves.
Check the cited content. A correct answer from the wrong article can mean Fin gives a different customer the wrong answer.
Test audience-targeted content with real customer profiles. If an article is only used for some customers, check the answer in Preview as a customer in that audience.
Keep your Eval. Save the Eval to Fin Main after you publish, so you can rerun it as a regression check after future changes.
FAQs
Does an Eval in a release test the release content or the live content?
Does an Eval in a release test the release content or the live content?
An Eval run inside a release tests the release's version of your content, guidance, procedures, and skills. Your live content on Fin Main isn't used, and no customers are affected.
Will customers see changes saved to a release?
Will customers see changes saved to a release?
Customers won't see changes saved to a release until the release is published. Until then, customers, Fin, and Copilot use the Fin Main version of your content.
Can Operator publish a release without my approval?
Can Operator publish a release without my approval?
Operator can't publish a release without your approval. Operator always summarizes what will change and asks you to confirm before it publishes a release or starts a rollout.
What can't I do in a release?
What can't I do in a release?
Some changes can't be made in a release:
Data Connector changes can't be added to a release by Operator.
Rolling back a published release can't be done by Operator. Use Roll back changes in the release instead.
How do I open a release after saving changes with Operator?
How do I open a release after saving changes with Operator?
To open a release after saving changes with Operator, click the release icon next to the message box, then click Open release. You can also click the In release label next to a saved change in the chat. Both open the release, where you can run Evals and publish.
Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts











