Skip to main content

Sync content from Amazon S3

Sync Markdown, PDF, DOCX, and HTML files from an Amazon S3 bucket into Fin, Copilot, and your Help Center, daily or one time.

Written by Beth-Ann Sher

Use this article to connect an Amazon S3 bucket to Fin, choose which files sync and what they become, and understand the limits of Amazon S3 content sync.

Amazon S3 content sync copies files from 1 Amazon S3 bucket into Fin, so Fin AI Agent, Copilot, and your Help Center can use them. You choose which folders and files to sync, and what each file type becomes: a public article, an internal article, a snippet, or a document. You can keep the content up to date with a daily sync, or import it once.

Note: Amazon S3 content sync is in open beta. It's available whether you use Fin with Intercom or with another helpdesk.


What you need before you sync content from Amazon S3

To set up Amazon S3 content sync, you need:

  • The bucket name. Each Amazon S3 source reads files from 1 bucket, which can be in any AWS region.

  • The AWS account ID of the account that owns the bucket. This is a 12-digit number.

  • Permission to edit the bucket policy in the AWS account that owns the bucket.

  • Permission to edit the AWS Key Management Service (KMS) key policy, only if your files are encrypted with a customer managed KMS key.

You don't need to create an AWS Identity and Access Management (IAM) role or an access key for Amazon S3 content sync. Fin uses its own read-only role, so no AWS credentials are stored in your workspace. You give that role access in your bucket policy, for this source only.

What permissions does the Amazon S3 bucket policy give Fin?

Fin generates the bucket policy statements for your Amazon S3 source during setup. This table shows the 4 read-only permissions the statements give Fin:

Statement

Permission

What it lets Fin do

List

s3:ListBucket

List the files in the bucket, or only the files under your prefix

Location

s3:GetBucketLocation

Find the bucket's AWS region

Read

s3:GetObject

Read the files

Tags

s3:GetObjectTagging

Read file tags, so you can filter on them

Note: The Amazon S3 bucket policy doesn't give Fin any permission to write, change, or delete files in your bucket.

This example shows the generated bucket policy statements for an Amazon S3 source that uses the prefix docs/. The values in capitals, such as YOUR_BUCKET and SOURCE_ID, are placeholders. Fin fills in the real values for your source during setup.

[
  {
    "Sid": "FinContentSyncListSOURCE_ID",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::FIN_ACCOUNT_ID:role/s3-content-sync-role" },
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::YOUR_BUCKET",
    "Condition": {
      "StringEquals": { "aws:userid": "FIN_ROLE_ID:SOURCE_ID" },
      "StringLike": { "s3:prefix": "docs/*" }
    }
  },
  {
    "Sid": "FinContentSyncLocationSOURCE_ID",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::FIN_ACCOUNT_ID:role/s3-content-sync-role" },
    "Action": "s3:GetBucketLocation",
    "Resource": "arn:aws:s3:::YOUR_BUCKET",
    "Condition": {
      "StringEquals": { "aws:userid": "FIN_ROLE_ID:SOURCE_ID" }
    }
  },
  {
    "Sid": "FinContentSyncReadSOURCE_ID",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::FIN_ACCOUNT_ID:role/s3-content-sync-role" },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::YOUR_BUCKET/docs/*",
    "Condition": {
      "StringEquals": { "aws:userid": "FIN_ROLE_ID:SOURCE_ID" }
    }
  },
  {
    "Sid": "FinContentSyncTagsSOURCE_ID",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::FIN_ACCOUNT_ID:role/s3-content-sync-role" },
    "Action": "s3:GetObjectTagging",
    "Resource": "arn:aws:s3:::YOUR_BUCKET/docs/*",
    "Condition": {
      "StringEquals": { "aws:userid": "FIN_ROLE_ID:SOURCE_ID" }
    }
  }
]

Without a prefix, the List statement has no s3:prefix condition, and the Read and Tags statements use the Amazon Resource Name (ARN) arn:aws:s3:::YOUR_BUCKET/* as their resource.

If your files are encrypted with a customer managed KMS key, Fin also generates 1 key policy statement for your Amazon S3 source:

{
  "Sid": "FinContentSyncDecryptSOURCE_ID",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::FIN_ACCOUNT_ID:role/s3-content-sync-role" },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:userid": "FIN_ROLE_ID:SOURCE_ID",
      "kms:ViaService": "s3.BUCKET_REGION.amazonaws.com"
    }
  }
}

The kms:ViaService condition means Fin can only use the KMS key to read files from Amazon S3 in your bucket's region.


How to sync content from Amazon S3

Step 1: Add Amazon S3 as a source

To add Amazon S3 as a content source, go to Fin AI Agent > Train > Content. Under "Add content", click From Amazon S3.

The Amazon S3 setup has 4 tabs: Connect, Filters, Target, and Review.

Step 2: Enter your Amazon S3 bucket details

On the Connect tab, under Bucket, enter your Amazon S3 bucket details:

  • Bucket name: for example, my-help-docs-bucket.

  • AWS account ID: the 12-digit ID of the AWS account that owns the bucket.

  • Prefix (optional): a folder path, for example docs/, to sync from 1 folder only. The prefix can't contain *, ?, or $.

Click Next to see your bucket policy.

Step 3: Add the bucket policy and verify access

After you enter your bucket details, Fin shows a Bucket policy section with the statements generated for your Amazon S3 source. To give Fin access to your bucket:

  1. Open your bucket in the Amazon S3 console and edit its bucket policy.

  2. In Fin, copy the Bucket policy statements. Click Show full policy to see all of them.

  3. Add the statements to the Statement list of your bucket policy.

  4. If your files use a customer managed KMS key, copy the Key policy statement from Fin too, and add it to the key policy of that key.

  5. Save the bucket policy.

  6. In Fin, click Verify.

Important: Add the generated statements to your existing bucket policy. Don't replace the whole bucket policy, because that removes any access it already gives to other users and services.

When the Verify check passes, Fin opens the Filters tab. To use a different bucket, click Change bucket and enter the new details.

Note: Verify needs at least 1 non-empty file in your bucket, or under your prefix, to test Fin's access. If the bucket or prefix is empty, add a file before you click Verify.

Step 4: Choose which files to sync and what they import as

On the Filters tab, under Choose files and file type mapping, choose which file types sync from Amazon S3 and what each one becomes in Fin:

  • File types: choose which file types to sync: Markdown, PDF, DOCX, and HTML. All 4 are selected by default.

  • File types mapping: in the Imports as column, choose what each file type becomes in Fin.

This table shows the Imports as options and the default for each file type in Amazon S3 content sync:

File type

Options

Default

Markdown

Public article, Internal article, Snippet

Public article

PDF

Public article, Internal article, Snippet, Document

Document

DOCX

Public article, Internal article, Snippet, Document

Document

HTML

Webpage

Webpage

Note: PDF and DOCX files converted to an article or snippet can lose some formatting, such as tables and images. To keep as much of the original formatting as possible, map them to Document.

Under Choose which files to sync, select folders and files in the Files to sync list:

  • To open a folder, click the arrow next to it.

  • To sync a whole folder, select it. Selected folders also include files added to them later.

  • To sync everything in the bucket, or everything under your prefix, select Select all.

  • To see more files in a large folder, click Load more.

The Files to sync list shows folders and only the file types you chose.

Step 5: Filter Amazon S3 files by metadata (optional)

Under Filter by file metadata, you can add rules to choose which of your selected Amazon S3 files sync. You can filter on these file fields:

  • Folder path

  • File name

  • Size (bytes)

  • Last modified

  • Tags (for example, owner=support)

Fin shows how many files match your rules.

Note: File counts during setup don't include tag rules, because Fin only reads tags when files sync. The number of files that actually sync can be lower than the count shown.

Step 6: Choose who can use the synced content

On the Target tab, under Choose content access, choose where Fin uses the content synced from Amazon S3:

  • Fin AI Agent: Fin uses the content to answer customers.

  • Copilot: teammates can use the content in Copilot.

  • Help Center: public articles are visible to customers in your Help Center. If you turn this on, select a help center.

You can also add audience rules to limit which customers Fin uses the synced content for. Click Next to review your settings.

Step 7: Choose a sync mode and finish

On the Review tab, check your Amazon S3 source settings under Review and finish. Then, under Sync mode, choose how the content stays up to date:

  • Ongoing sync: keeps the content up to date with the bucket every day.

  • One-time import: copies the files once. Later changes in the bucket aren't synced.

Click Sync and publish for an ongoing sync, or Import and publish for a one-time import. Fin starts reading your files, and you'll get an email when the sync finishes.


How to re-sync, edit, or remove an Amazon S3 source

To manage an Amazon S3 source, go to the source and open the Manage menu. You can choose:

  • Re-sync: reads the bucket again now. Fin skips files that haven't changed.

  • Edit source: changes the file selection, filters, content access, and sync mode.

  • Remove this source: removes the source and all its synced content, including folders and subfolders, from your workspace.

Note: The Manage menu options aren't available while the Amazon S3 source is syncing.

Important: Removing an Amazon S3 source deletes all of its synced content from your workspace, and you can't undo it. Your files in Amazon S3 aren't affected.

What can't you change when you edit an Amazon S3 source?

When you edit an Amazon S3 source, some settings can't change:

  • Bucket, account ID, and prefix: to use a different bucket, remove the source and create a new one.

  • File type mapping: this can't change after the first import. To change it, remove the source and connect it again.

  • Help center: the help center chosen during setup can't change.

  • Content access: changes apply to newly synced content only. Existing content keeps its current settings.


FAQs

Which file types does Amazon S3 content sync support?

Amazon S3 content sync supports Markdown (.md), PDF (.pdf), Word (.docx), and HTML (.html, .htm) files. Other file types, such as .doc, don't show in the Files to sync list and don't sync.

A PDF, DOCX, or HTML file is also skipped when its Content-Type metadata in Amazon S3 doesn't match its file extension. For example, a PDF stored with the Content-Type text/html doesn't sync.

What is the maximum file size for Amazon S3 content sync?

Amazon S3 files over the maximum size for their type are skipped and don't sync. This table shows the maximum file size for each file type:

File type

Maximum size

PDF, DOCX

100 MB

HTML

16 MB

Markdown

5 MB

PDF and DOCX files mapped to an article or snippet have a second limit: their converted text must be 5 MB or less, or the file is skipped.

Note: If an Amazon S3 file that has already synced grows past its maximum size, the sync fails until you make the file smaller or remove it from the selection.

How many files can an Amazon S3 source sync?

One Amazon S3 source syncs up to 100,000 files. If your selection has more, Fin syncs the first 100,000 and doesn't remove content already synced from the rest. To stay under the limit, use a prefix, folders, or filters to narrow your selection, or create more sources.

Can you sync one Amazon S3 bucket to more than one source or workspace?

Each Amazon S3 source reads 1 bucket. You can create several sources on the same bucket in one workspace, but their prefixes can't overlap.

An Amazon S3 bucket can belong to only 1 workspace, even when 2 workspaces belong to the same company. If another workspace already uses the bucket, Verify shows "Another Fin workspace already uses this bucket. Use a different bucket." The bucket becomes available again when the last source on it is removed.

Which encryption types does Amazon S3 content sync support?

Amazon S3 content sync reads files encrypted with Amazon S3 managed keys (SSE-S3) or with a customer managed KMS key. Fin can't read files encrypted with the AWS managed key (aws/s3). To sync those files, encrypt them with SSE-S3 or a customer managed key instead.

What happens when you move or rename a file in Amazon S3?

When you move or rename an Amazon S3 file inside your selection without changing its contents, Fin keeps the same synced item and updates its title and folder. Sometimes AWS gives the moved file a new fingerprint, for example when a large file is copied in parts. In that case, Fin adds the file as a new item and removes the old one.

How many folder levels does Amazon S3 content sync copy?

Amazon S3 content sync copies your folder structure up to 10 levels deep. Files in deeper folders go into the deepest copied folder.

Which language and collection do public articles synced from Amazon S3 use?

Public articles synced from Amazon S3 use the language of the help center you chose. They aren't added to a collection.

Where is content synced from Amazon S3 stored?

Your Amazon S3 bucket can be in any AWS region. Fin stores the synced content in your workspace's region.


💡Tip

Need more help? Get support from our Community Forum
Find answers and get help from Intercom Support and Community Experts


Did this answer your question?