{"id":22125,"date":"2019-11-05T12:13:18","date_gmt":"2019-11-05T12:13:18","guid":{"rendered":"https:\/\/www.intercom.com\/blog\/?p=22125"},"modified":"2020-07-30T12:54:17","modified_gmt":"2020-07-30T11:54:17","slug":"updating-our-user-storage","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/updating-our-user-storage\/","title":{"rendered":"Making our user storage more scalable and secure"},"content":{"rendered":"

Our mission is to make web business personal. We want our customers to be able to have the same delightful, personal experiences talking to their customers as they do talking to their friends.<\/p>\n

Enabling all these conversations and interactions between businesses and their customers requires an extremely robust storage system that can scale as the data set expands \u2013 our storage needs, after all, have to keep pace not just with our own growth, but with the combined growth of all our customers.<\/p>\n

In our system, we use the term \u201cusers\u201d to describe the people who interact with our customers via Intercom, and needless to say, that means storing a very extensive live list of users.<\/p>\n

“Managing user storage is the very foundation of the Intercom platform, and has required constant and rapid evolution since we started the company”<\/p><\/blockquote>\n

Managing this user storage is the very foundation of the Intercom platform<\/a>, and has required constant and rapid evolution since we started the company. Recently, we completed an ambitious transition to an entirely new user storage system \u2013 the story of how our user storage has evolved from our earliest days to now shines a light on how Intercom itself has evolved and scaled as a platform and as a company over the past eight years.<\/p>\n

A brief history of user storage<\/h2>\n

2011 – MySQL<\/h3>\n

In the beginning we used a single Amazon RDS (relational database) instance for everything \u2013 including user data.<\/p>\n

When we launched custom data for users, we opted to serialize it in a MySQL TEXT field.<\/p>\n

This solution worked until we wanted to improve the user list by allowing customers to sort and filter on custom data, at which point we needed a more flexible solution.<\/p>\n

2011-2014 – MongoDB<\/h3>\n

The solution we turned to was MongoDB<\/a>, which was amazingly flexible for our purposes. It allowed us to easily store custom data and other complex data structures. We implemented a very flexible query pattern to power the user list. For our early years, this was more than adequate.<\/p>\n

However, things that worked at low scale started to creak as we grew \u2013 we encountered problems powering the user list with MongoDB, and began to experience timeouts for some customers. It became a challenge for the database to serve a high volume of “single user reads\/writes” as well as the search workload, and it became clear that we urgently needed to upgrade.<\/p>\n

2014-2017 – MongoDB and Elasticsearch<\/h3>\n

In response, we started streaming user data out in to Elasticsearch<\/a> and used it to power the user list. Around the same time, MongoDB itself got a lot more stable.<\/p>\n

That period of stability was not to last \u2013 in late 2016 MongoDB really started creaking for our purposes. We experienced regular outages (sometimes daily) and the constraints on large customers became untenable.<\/p>\n

Internally, too, we were suffering \u2013 our infrastructure team spent nearly all their time keeping user storage alive, and it was not the sort of problem that we could solve by spending more with AWS.<\/p>\n

Time to make a change<\/h2>\n

The situation was unsustainable and extremely stressful for the team \u2013 folks who were on call had to take their laptop with them everywhere, even to parties. That constant state of anxiety that something might break at any moment takes a serious toll, and it was clear something needed to change.<\/p>\n

“To support our new scale, we needed to fundamentally change our user storage layer”<\/p><\/blockquote>\n

To support our new scale, we needed to fundamentally change our user storage layer.<\/p>\n

The options appeared to be either:<\/p>\n