{"id":25903,"date":"2021-05-19T18:00:08","date_gmt":"2021-05-19T17:00:08","guid":{"rendered":"https:\/\/www.intercom.com\/blog\/?p=25903"},"modified":"2021-05-19T18:10:19","modified_gmt":"2021-05-19T17:10:19","slug":"core-technologies-team","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/core-technologies-team\/","title":{"rendered":"Core responsibility: How we scaled our core technologies team"},"content":{"rendered":"

It\u2019s a familiar problem for all companies that scale fast \u2013 how do you keep your core technologies manageable for the increasing number of teams that depend on them?<\/p>\n

As Intercom\u2019s product matured, our product teams needed to expand the depth and breadth of their technology expertise in order to support the features we were building for our customers.<\/p>\n

This surfaced as an increasingly large percentage of our product teams\u2019 time being spent on operations or deep diving into understanding our small set of core technologies<\/a>. How our core technologies were being used was also diverging, leading to needlessly increased complexity in our systems. Our product teams were being slowed down and our operational health was taking a hit.<\/p>\n

“This is the story of how we grew and expanded our team over the past few years as the company rapidly scaled, and how we have gone about expanding our responsibilities”<\/p><\/blockquote>\n

Once we reached a certain size, we could justify devoting a small team to focus exclusively on enabling teams to more efficiently use the technologies we build Intercom on. So in late 2018, Team Datastores was created, initially with a fairly conservative area of responsibility.<\/p>\n

It wouldn\u2019t be an operations team, as such \u2013 the product teams would still own their infrastructure. Instead, it would be an internal core technologies team that would take long-term views on how we build and scale Intercom. This is the story of how we grew and expanded our team over the past few years as the company rapidly scaled, and how we have gone about expanding our responsibilities.<\/p>\n

Our first challenge \u2013 Elasticsearch<\/h2>\n

As we formed and figured out where to start, we had an obvious first technology: Elasticsearch<\/a>, which was originally introduced<\/a> in order to support advanced search capabilities around custom data attributes and is the only datastore we fully manage.<\/p>\n

Over the subsequent years, as Intercom became a more powerful product, Elasticsearch was used to enable more features. Product teams typically set up and owned their own Elasticsearch infrastructure<\/a> with little guidance, technical depth, or use of best practices.<\/p>\n

This resulted in each cluster being its own unique snowflake running on different hardware, different versions of Elasticsearch, and different configurations. Product teams were spending an increasing amount of time running Elasticsearch instead of solving customer problems. Our general availability around Elasticsearch also suffered and the learnings from availability incidents were not being applied to all Elasticsearch clusters.<\/p>\n

Build a map of the problem space<\/h2>\n

To improve any situation, you first need to know the exact situation you\u2019re in.<\/p>\n

As we formed the team, we were very conscious of only taking on the most impactful technology slice we could and building ownership, stability, and momentum around it before taking on another area. We hoped this would build confidence within the team and provide evidence to our stakeholders for further investment.<\/p>\n

To achieve this with Elasticsearch, we worked through these steps:<\/p>\n

<\/p>\n

    \n
  1. Understand and document how we are currently using Elasticsearch. Build the necessary technical depth within the team needed through training and experimentation.<\/li>\n
  2. Using our learnings, define what the gold standard for Elasticsearch is in Intercom.<\/li>\n
  3. Roll that gold standard out to all Elasticsearch clusters; rationalize where possible.<\/li>\n
  4. Automate the process of achieving and staying at the gold standard.<\/li>\n
  5. Increase observability and empower teams to support their infrastructure through tooling and training.<\/li>\n
  6. Move further up the technology stack and provide cross-org solutions for problems that multiple teams face. For example: AMI rollouts, node replacement, and security patching.<\/li>\n<\/ol>\n

    The positive impact we had on engineering was readily apparent, which led to more investment in the core technologies area. It also led to a queue of engineers wanting to join the team.<\/p>\n

    Widening the circle of ownership<\/h2>\n

    Over that year, Intercom\u2019s use of Elasticsearch matured<\/a> to a point where we were ready to take on another technology. Stronger support around RDS Aurora<\/a> was top of mind as one of our MySQL tables was already far outside of AWS\u2019 best practices. This spawned our sharding program<\/a> and our team taking ownership of our largest RDS Aurora databases. Again, we followed the same maturity trend but had the benefit of building on top of a managed service. This time, much of the \u201cgold standard\u201d setup we got for free from RDS Aurora.<\/p>\n

    “This prompted us to step back and consider the principles we\u2019d use to determine whether to expand our footprint to include new core technologies”<\/p><\/blockquote>\n

    With our success around Elasticsearch and RDS Aurora, we had the momentum to further grow the size of the team and take on more of Intercom\u2019s core technologies. This prompted us to step back and consider the principles we\u2019d use to determine whether to expand our footprint to include new core technologies or mature the technologies we already owned.<\/p>\n

    Criteria for expanding ownership<\/h2>\n

    How did we decide to take on a new technology versus further mature something we already owned?<\/p>\n

    We arrived at these key criteria:<\/p>\n