Skip to main content

How to update content and test it in a release before publishing

Update content with Fin Operator, save it to a release, test it with Evals, and publish every change together.

Written by Dawn P

Use this article to update your content with Operator, save the changes to a release instead of publishing them straight away, test how Fin answers with those changes using Evals, and then publish every change together. This process is useful when a change affects several articles at once, such as a pricing or policy update with a fixed go-live date.

Note: Operator is included as part of the Pro add-on. You'll need the Pro add-on to use Operator.

What does the release testing process look like?

The release testing process lets you prepare content changes, test them with Fin, and publish them together, without changing what customers see on Fin Main (the live version of your content that customers, Fin, and Copilot use) until you're ready. A release is a saved set of content changes that you test and publish together.

The process follows 5 steps:

  1. Update your content with Operator, or edit it yourself in Fin Main.

  2. Save the changes to a release draft instead of publishing them.

  3. Create an Eval with Simulations that cover the questions your customers will ask.

  4. Run the Eval on the release, and on Fin Main to compare.

  5. Publish the release when the results look right.

For example, say your refund window is changing from 30 days to 14 days on a set date. You can update every article that mentions the refund window, save the changes to one release, check that Fin gives the new answer, and publish them all on the day the policy changes.


How to update content with Operator

Operator can find every article affected by a change and propose updates for you to review. To update content with Operator:

  1. Open Operator.

  2. Describe the change and ask Operator to find and update the affected content. For example: "Our refund window is changing from 30 days to 14 days. Find every article and snippet that mentions the refund window and propose updates."

    A new Operator chat with a prompt typed in the message box: the refund window is changing from 30 days to 14 days, and Operator should find every article that mentions the refund window and propose updates.
  3. Review each proposal in the review panel. You can see what's being added, removed, or changed before anything is saved.

  4. Ask Operator to adjust any proposal that isn't quite right.

Tip: You can also edit an article yourself in Fin Main and save it to a release the same way. Learn more in how to publish, save, and unpublish articles in Fin Main.


How to save content changes to a release

Saving content changes to a release keeps them out of Fin Main, so customers and Fin keep using the current content until you publish the release. To save Operator's proposed changes to a release:

  1. In the review panel, click the arrow next to Publish to main.

  2. Choose an existing release, or click New release draft.

    The Operator review panel with 2 changes pending and the Publish to main menu open. Under Save to release draft it lists existing releases and a New release draft option, which an arrow points to. The article on the right shows the change from 30 days to 14 days.
  3. For a new release, enter a name and an optional description, then click Create release.

    The Create a release dialog with a Name field filled in as Refund window update, an optional Description field, and Cancel and Create release buttons.
  4. Repeat for each proposal you want in the same release.

After you save a change, a release icon with a count appears next to the Operator message box. Click the icon to see the release name, its status, and the items in it, then click Open release to go to the release.

Operator chat showing 'You applied 2 changes' with two articles, How to request a refund and Processing Refunds at Examply, each marked In release. An arrow points to the In release label. The release icon with the number 2 sits below the message box.

You can also click the In release label next to a saved change in the chat to open the release.

The release summary opened from the release icon next to the message box. It shows the release Refund window update with the status Draft and 2 edited, lists both changed articles with their changes, and has an Open release button at the bottom. An arrow points to the release icon.

Note:

  • Changes saved to a release aren't live. Customers, Fin, and Copilot keep using the Fin Main version until you publish the release.

  • If you edit an item directly inside a release, the editor doesn't save automatically. Click Save to release often, or you'll lose your edits if you refresh the page.


How to create and run an Eval for a release

An Eval is a set of saved test conversations, called Simulations, that Fin answers so you can check its responses. When you run an Eval inside a release, Fin answers using the release's version of your content, so you can see how your changes perform without affecting any customers. Each Eval can hold up to 50 Simulations.

How to create an Eval for a release with Operator

  1. Ask Operator to create an Eval for your change. For example: "Create an Eval called Refund window update with Simulations for a customer asking how long they have to request a refund, a customer asking for a refund after 20 days, and a customer asking whether the policy has changed. Save it to the release draft Refund window update."

    A new Operator chat with a prompt typed in the message box: create an Eval called Refund window update with simulations for a customer asking how long they have to request a refund, a customer asking for a refund after 20 days, and a customer asking whether the policy has changed, and save it to the release draft Refund window update.
  2. Review the Simulations Operator suggests in the changes pending panel above the message box. Each Simulation is one test conversation, such as a customer asking about the new refund window. Ask Operator to add, remove, or change any Simulation before you save.

    Operator's reply after creating the Eval, with 3 changes pending: the simulations Refund policy changed, Refund after 20 days, and Refund window length, each marked Apply to release. The Save to release button is at the bottom of the panel. The open simulation, Refund policy changed, shows the chat channel, the customer's first message 'Has your refund policy changed recently?', and checks for no handoff and a Fin reply.
  3. Click Save to release. All the Simulations are saved to the Eval in your release.

  4. Run the Eval from the release's Evals tab, click Run eval on the Eval's row. Every Simulation in the Eval runs using the release's version of your content.

    The Evals tab of the release Refund window update, which is in Draft status. Under Changed in this release it lists the new Eval Refund window update, with 3 simulations not run, and a Run eval button. Below it, From Fin Main lists 21 Evals, and an Add more evals option sits at the bottom.
  5. Open the results to check each Simulation's answer and the content Fin used to answer it.

    The results page for the Refund window update Eval after it has run in the release. Simulation results show 3/3 passed. The simulated conversations Refund policy changed, Refund after 20 days, and Refund window length each show a Pass result and a last run of Just now. A Run all button sits above the list.

If a Simulation fails, update the content in the release, then run the Eval again. To edit an item in a release, open the release, click the item, click Edit content, make your changes, and click Save to release.

Tip: Use Preview in the release to chat with Fin as a customer would. Preview conversations use the release's changes, don't touch Fin Main, and aren't charged.


How to publish a release

Publishing a release makes every change in it live on Fin Main at the same time. To publish a release:

  1. Open the release and check that your Evals have passed.

  2. Click Release at the top right of the release page.

    The Refund window update release with the Release menu open at the top right. The menu offers Publish to Fin Main, which publishes changes to Fin Main and releases them to all customers, and Roll out gradually, which releases to a share of customers before it goes to everyone. The Evals tab shows the Refund window update Eval with 3/3 Passed.

    You can also click Release under the release name in the left sidebar.

    The Refund window update release in Draft status with the Release button under the release name in the left sidebar. The same menu opens beside it, with Publish to Fin Main highlighted. The Evals tab shows the Refund window update Eval with 3/3 Passed.
  3. Select Publish to Fin Main. To release the changes to a share of customers before everyone, select Roll out gradually instead.

After you publish, the release shows the status Published, and customers, Fin, and Copilot use the updated content straight away. You can also ask Operator to publish the release. Operator summarizes what will change and asks you to confirm before it publishes.

Important: A published release can't be undone as a whole. To reverse changes after publishing, use Roll back changes in the release. Learn more in how to roll out a release in Fin.


What are the best practices for testing content in a release?

These best practices help you get reliable results when you test content changes in a release:

  • Keep one change per release. Group the content for one change, such as a policy update, in its own release so you can test and publish it on its own.

  • Run a baseline first. Run your Eval on Fin Main before you make changes, so you can see what the release improves.

  • Check the cited content. A correct answer from the wrong article can mean Fin gives a different customer the wrong answer.

  • Test audience-targeted content with real customer profiles. If an article is only used for some customers, check the answer in Preview as a customer in that audience.

  • Keep your Eval. Save the Eval to Fin Main after you publish, so you can rerun it as a regression check after future changes.


FAQs

Does an Eval in a release test the release content or the live content?

An Eval run inside a release tests the release's version of your content, guidance, procedures, and skills. Your live content on Fin Main isn't used, and no customers are affected.

Will customers see changes saved to a release?

Customers won't see changes saved to a release until the release is published. Until then, customers, Fin, and Copilot use the Fin Main version of your content.

Can Operator publish a release without my approval?

Operator can't publish a release without your approval. Operator always summarizes what will change and asks you to confirm before it publishes a release or starts a rollout.

What can't I do in a release?

Some changes can't be made in a release:

  • Data Connector changes can't be added to a release by Operator.

  • Rolling back a published release can't be done by Operator. Use Roll back changes in the release instead.

How do I open a release after saving changes with Operator?

To open a release after saving changes with Operator, click the release icon next to the message box, then click Open release. You can also click the In release label next to a saved change in the chat. Both open the release, where you can run Evals and publish.


💡Tip

Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts


Did this answer your question?