How Intercom ensures data privacy and safety in the age of AI

Generative AI has transformed customer service, with AI agents leveraging large language models (LLMs) to understand and resolve customer queries.

While this significant change has brought with it multiple benefits for support teams and customers, it has also come with new data security and privacy risks. In the age of AI, handling customer data correctly isn’t just a regulatory requirement, it’s the foundation of trust between you and your customers.

At Intercom, we’ve designed our AI-powered products, such as our AI agent, Fin, with data protection as a first principle, implementing multiple layers of security controls, accuracy checks, and privacy safeguards that exceed industry standards.

Here, we take a transparent look at everything we’re doing to ensure our customers’ complete peace of mind.

Protecting customers’ privacy

Ensuring our customers trust us to handle their data safely has always been a core part of our philosophy. In this new AI era where we’re using third-party service providers to build AI-powered products and features, robust protections and compliance protocols are more important than ever.

While this introduces additional complexity, we’re working hard to ensure it doesn’t impact our customers’ experience. Here’s how we’re keeping our customers safe while minimizing any red tape they have to deal with.

• Hosting infrastructure tailored to our global customer base: To ensure customers around the world can securely make use of our AI features, we’ve introduced new hosting infrastructure options. Most notably, our European customers can now access Intercom’s AI-powered products and features, including Fin AI Agent, Copilot, and inbox AI features, with data processed securely within Europe for enhanced data privacy. Existing European customers who were previously using US data processing will be automatically transitioned to the new EU-hosted infrastructure.
• Constant evaluation of third-party service providers: To provide the best possible customer experience, we’re committed to only working with the best third-party service providers. This means we’re constantly testing and evaluating which providers are performing best and returning the most accurate, high-quality results. Metrics we test for include resolution rates, hallucination rates, and latency in responses, amongst others. For the most up-to-date list of third-party service providers we’re working with, take a look here.
• Careful restrictions when it comes to data handling: None of our third-party LLM providers store any conversation data. Customer data is only temporarily used to generate responses before being deleted. In addition, all third-party LLM providers are contractually prohibited from using conversation data to train or improve their models. Access to our customers’ data is strictly limited to what’s necessary for providing the relevant service.
• Robust contractual protections to ensure compliance: Business Associate Agreements (BAAs) are in place with all third-party LLM providers we use for HIPAA-compliant customers. We also require these providers to implement appropriate technical and organizational measures to protect personal data to the standard set out by the applicable Data Protection legislation.

Mitigating LLM-focused threats

Data security is more important than ever when working with AI because, unlike traditional software systems where threats are often well-documented and predictable, LLMs can be vulnerable in ways that are still being discovered and understood by the security community.

To stay ahead, our security teams are always thinking beyond conventional security measures and anticipating potential vulnerabilities before they’re exploited, continuously monitoring for new types of attacks, and maintaining security systems that can adapt as rapidly as the threats themselves evolve.

Here are a few of the most important threats, identified in the OWASP LLM Top 10 threats list, that could put customers at risk and the steps we’ve taken to actively secure our AI-powered products against them.

Direct prompt injection

Direct prompt injection involves malicious actors attempting to manipulate the LLM supporting an AI agent by injecting harmful or misleading prompts. These attacks can potentially cause the AI agent to generate unintended or harmful responses.

While prompt injections can’t be universally prevented, we’ve applied a series of guardrails to reduce the likelihood of prompt injection exploitation and eliminate any real business impact if a prompt injection does occur. The guardrails we’ve put in place include:

• Defensive prompting: We include protective instructions whenever the LLM reads user input, safeguarding against attempts to manipulate or override the system’s core safety measures.
• Monitoring and adaptation: We track the emergence of new prompt injection techniques and any instances where Fin’s AI features behave in unexpected ways. We then modify our defensive prompting strategy accordingly to counteract any new threats.
• Version upgrades: We assess and migrate to newer versions of our LLM providers’ generative models, which are increasingly robust against prompt injection attacks.
• Rapid response: Over the past year, we’ve observed very few successful attempts at bypassing the safeguards put in place to protect our AI-powered features. None of these attempts constituted a reputational threat to our customers, and all incidents were mitigated in a timely fashion.

Data poisoning

Data poisoning involves the injection of malicious data into the training set.

We’ve secured against this happening by taking these steps:

• Data validation: We’ve put rigorous data validation protocols and platform controls in place to ensure the integrity and quality of Knowledge Base data.
• Limiting context: In each LLM prompt, we only include the context that the LLM needs to perform a specific operation. This reduces the likelihood of incorporating any poisoned data at scale.

Evasion attacks

Evasion attacks involve adversaries crafting inputs that bypass AI features’ security measures and produce harmful outputs.

Our mitigation strategies to avoid this include:

• Adversarial testing: We run industry-standard LLM vulnerability scanning against the models we use in production to identify and mitigate potential evasion techniques.
• Robust prompt design: We’ve invested a significant amount of time in designing our prompts to be robust against manipulation, ensuring Fin’s outputs remain consistent and secure.
• Regular updates: Security measures are regularly updated to address new evasion techniques as they emerge.

Insecure integrations

Insecure integrations involve vulnerabilities arising from the integration of the LLM or AI features with other systems or applications.

To ensure an ironclad integration, we’ve implemented the following:

• Secure API design: APIs used to integrate the LLM with Intercom and/or other systems are designed with security best practices in mind, including secure defaults like secure proxy to LLM providers and complementary customer control implementation for specific features.
• Penetration testing: Regular penetration testing is conducted to identify and address integration vulnerabilities.
• Security reviews: Comprehensive security reviews are performed for all integrations to ensure they meet our stringent security standards.

Data leakage

Data leakage occurs when sensitive or confidential information is inadvertently exposed through AI-powered features’ responses.

To protect against this, we’ve employed the following strategies:

• Granular opt-in: We design our AI features in such a way that customers must explicitly opt in to making data sources available for LLM processing. This means customers have fine-grained control of the content they’re comfortable feeding to AI-backed features.
• Data leak prevention by design: We operate under the assumption that the data going into an LLM prompt can come out as part of the LLM response. All features that surface an LLM response ensure that the data being fed to the LLM is already authorized to be read by the relevant parties involved.
• Access controls: Strict access controls are enforced to limit the exposure of sensitive data within the LLM’s operational environments. Intercom’s LLM providers implement a zero data retention policy on customer messages and returned responses, and we mandate that no data is used for training or fine-tuning of their LLMs.
• Auditing and logging: Comprehensive logging and auditing mechanisms are in place to track data access and usage, ensuring any anomalies are quickly identified and addressed. Prompts are logged in access controlled repositories and access is gated according to business needs. Our LLM providers are considered key sub-processors and are therefore subject to periodic User Access Reviews and annual security audits as part of our risk management and compliance efforts.

Maximizing protection with performance assurance at every phase of the development cycle

Our commitment to delivering exceptional quality is reflected in our AI products’ performance, which consistently surpass publicly available LLMs.

Post-deployment, our AI-powered products are continuously monitored in production to ensure ongoing excellence. Our engineering and ML teams collaborate closely on rigorously testing every model and prompt modification, reviewing metrics, and analyzing customer reports to quickly address any emerging issues.

Backtesting

Before any updates are introduced, they undergo thorough scrutiny in our secured development environment. This ensures enhancements in key performance areas without compromising our AI products’ reliability. Our backtesting process includes:

• Minimizing hallucinations: We ensure that responses are grounded in the provided context, rather than being influenced by the LLM’s pre-existing knowledge.
• Ensuring quality: Both automated and manual quality checks are conducted to maintain the high standard of Fin’s responses.

A/B testing

We deploy each model or prompt change to a controlled subset of requests to evaluate:

• Maximizing resolution rates: This ensures that the updates improve the accuracy and usefulness of responses.
• Customer satisfaction: Assessing feedback and CSAT scores allows us to validate the positive impact of changes.
• Comprehensive performance assessment: Conducting large-scale evaluations helps to ensure sustained performance and quality.
• Optimizing latency and performance: We rigorously test for improved responsiveness without sacrificing accuracy.

Red teaming

To safeguard against vulnerabilities, we conduct extensive internal penetration testing, including:

• Guardrail testing: This involves regular simulations of real-world attack scenarios that incorporate the broad functionality of the Intercom platform to validate the robustness of our defenses.
• Hallucination assessments: We perform systematic evaluations of the effects a hallucination may have on downstream product behavior, ensuring a defense-in-depth approach when handling the LLM responses. Our high conversation volume has enabled us to build a substantial dataset of real-world hallucinations, which we use to validate our models against. This real-world testing capability, available only to established players like Intercom that processes millions of conversations, provides an additional layer of quality assurance.
• Vulnerability scanning: We apply industry-standard LLM vulnerability scans to identify and mitigate potential weaknesses. If a model fails to meet our stringent security criteria, it isn’t deployed.

Regular auditing

Insufficient auditing can lead to undetected security incidents and non-compliance with regulatory requirements. To guard against this, we perform the following:

• Comprehensive logging: Detailed logging of all LLM interactions is maintained to provide a clear audit trail.
• Regular audits: Regular security audits are conducted to ensure compliance with regulatory requirements and internal policies.
• Performance monitoring: Key operational metrics are continuously tracked, including resolution rates, deflection rates, user feedback (both positive and negative), distribution of response types and outcomes, and response characteristics like length and complexity.
• Incident response: An incident response plan is in place to quickly address any security incidents identified through auditing.

The future of customer service depends on trust

As AI continues to reshape customer service, trusting the providers you work with is critical. And that trust can’t be built on promises alone – vendors need to walk the walk and demonstrate a consistent commitment to ensuring data safety.

This space is moving fast, and we’re adapting and learning all the time. At Intercom, we’re committed to leading the way in setting the highest standards of trust when it comes to privacy and security.