Building an API for powerful customer data analysis

Today we’re delighted to launch our brand new API version 2.0, which makes it easier for our customers to pull, analyze, and update Intercom contact and conversation data. This will more easily allow our customers to transform their data into valuable performance reports and business insights.

It consists of an overhauled Conversations API, as well as brand new Contacts and Data Attributes APIs. Now you can filter contacts and conversations using any combination of fields, like contact location or conversation tag, to pull the exact data you want. Once you’ve updated your API version, you can systematically tag conversations, as well as leverage newly exposed contact attributes, more granular conversation timestamps, and out-of-the-box conversation metrics.

“We rebuilt our search capability in the new API to be more robust, intuitive and powerful in this release”

The benefits of this release are immense and enable you to improve your team’s performance with custom reports, drive decision-making by leveraging your customer data in business intelligence tools, and deliver deeper context for more personalized customer conversations using your Intercom data.

The area of search, in particular, has been transformed by this release, and I’d like to dive into how we rebuilt our search capability in the new API to be more robust, intuitive and powerful.

Finding the personal in the data

We often talk about our mission of making internet business personal – how we want to help companies foster better relationships with their customers through powerful, thoughtful technology.

Online businesses might not get to meet their customers face to face in the way a coffee shop owner or corner store retailer do, but internet businesses can forge a sense of connection in other ways by leveraging data to offer more personalized, useful interactions. In order to support your customers effectively, you need to know who their customers are, and how they make their decisions.

“If the data isn’t easily searchable and usable, it can effectively come between a business and its customers, obscuring the real people we’re trying to connect with”

However, key to this is having appropriate context – and at a large enough scale and with enough data, it’s possible to lose the person in the numbers, so to speak. If the data isn’t easily searchable and usable, it can effectively come between a business and its customers, obscuring the real people we’re trying to connect with. Having the right tools in place to manage, access, and analyze that data is critical to getting the right context on your customers, and without that context, relationships are incredibly hard to build.

We’ve seen this ourselves at Intercom – as we deal with more and more data, search becomes an increasingly fundamental problem. Accurate, robust, usable search is what makes all that data an asset rather than a hindrance.

For our customers, it was increasingly vital that we supplied them with the tools to quickly and intuitively search their customer data. Our existing API was showing its age, limiting how our customers could search details about their own customers and conversations. It was also clear that, as the number and variety of our customers has grown dramatically over the years, we needed to devise a search capability that was flexible enough for any imaginable use case, while retaining intuitive discoverability.

Searching for the right answers

Given the scale of the overhaul we were planning, we knew there was significant room to improve. For instance, our previous user filtering was limited to email address, and our User and Conversation list APIs only allowed ordering by a small number of attributes.

Our APIs were not RESTful or industry standard, and paging through lists of users was subpar.

As we looked to develop our APIs we considered the huge number of jobs that people use Intercom for, including jobs we might not yet be able to predict, and concluded that we should build a very versatile and intuitive search function into the API.

We developed a set of principles for our APIs, and how we want them to behave. They should be:

• Simple: Easy to grasp
• Usable: Easy to use
• Adaptable: Flexible enough for a variety of use cases
• Discoverable: Easy to perceive

Building a boring but innovative solution

As we adhere to the “build boring software” mantra, we looked at tried and trusted technology to help us achieve this. Intercom has long relied on Elasticsearch as our primary way of searching data for internal use, as well as using it to power some of our product features (such as Inbox Search). We have also grown an increasingly sophisticated team and infrastructure to help manage our numerous Elasticsearch deployments. Our growing expertise in the area made it the right choice to power our search in the new API.

Our challenge was to make the interface powerful yet intuitive, so we explored a number of options:

1: SQL-like language

One of our early explorations was on an SQL-like language, such as that used in Salesforce and Jira APIs. In practice, it would have looked something like this:

SELECT Id, Name
FROM Contact
WHERE Name = 'Sandy'

While this would have been extremely powerful, we quickly realized it would certainly not have been simple or usable. It would also have required a user to learn all the different syntax and language features to be able to use it effectively. Also, it was not clear how to deliver the SQL query in a Restful request.

2: Query string-based search language

Another option that seemed quite promising was a query string-based approach, such as that used by Zendesk’s API until recently. It would have looked like: query=updated_at:1579273187 state:open, where the number refers to the epoch or Unix date.

The most obvious drawback is that such a language would not have allowed full Boolean AND/OR queries, so would not have been powerful enough. Also, it would have required the customer to URL encode their queries, making it less readable and therefore not as usable.

3: JSON-based query language

After much experimentation, we landed on a JSON-based query language, made up of discrete composable blocks. This seemed to fulfill all the principles we established at the outset.

• Adaptable: it allows for very powerful full Boolean search queries that are easy to construct.
• Usable: It consists of straightforward composable blocks.
• Simple: Once you know the operators and basic building blocks, you can construct any query you like, with minimal effort.
• Discoverable: It uses JSON payloads, which are easy for humans to read, machines to parse, and are widely used.

Devising the building blocks

Once we had decided on this direction, we came up with a simple yet composable query language. The main building blocks are query objects containing a field, operator and value:

{
field: "The field on the Contact object that you wish to target",
operator: "The comparison operator",
value: "The value to compare to. Can be one of the supported Data Types"
}

The accepted operators are:

In the case of logical composite operators (AND and OR), the field is omitted, and the value is an array of query objects:

{
operator: "AND",
value: ["Array of query objects"]
}

This allows you to construct queries such as: “Get me all conversations that have been updated last week, are currently closed, and were assigned to Tier 1, Tier 2, or VIP inboxes.” This looks like the following:

Another example might be a search for: “Get me all contacts created this month who have lead_score over 80, located in United States, with phone number not empty.’ Such a search would look like this:

Achieving impact with context

As you can see, this language allows for complex, Boolean searches while adhering to our initial principles – crucially, it is robust and powerful enough for the infinite variety of jobs that you want to do with your data inside Intercom.

By focusing on making the API highly usable, adaptable and discoverable, we believe we provide that context with more impact than ever before. After all, it is that critical context that makes all the difference between knowing your customer, and obscuring them behind the data.