{"id":14384,"date":"2018-01-22T16:55:33","date_gmt":"2018-01-22T16:55:33","guid":{"rendered":"http:\/\/blog.intercom.com\/?p=14384"},"modified":"2020-07-30T12:57:05","modified_gmt":"2020-07-30T11:57:05","slug":"building-autonomously","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/building-autonomously\/","title":{"rendered":"Building autonomously: Improving our production systems in small steps"},"content":{"rendered":"

At Intercom, we have a few values that underpin our engineering culture \u2013 moving fast but optimizing for the long term; doing less but doing it better; and taking ownership of our areas of responsibility and the things we build.<\/p>\n

These values allow us to be more autonomous as engineers because they provide a structure for us to make better decisions. When you\u2019re used to working with them all the time, they can fade into the background a bit, but they become especially useful when working in isolation from your team.<\/p>\n

A few months ago, I was visiting Chicago for personal reasons and so would be working away from my team for two weeks. There wouldn\u2019t be other engineers in my timezone for most of the day, so picking a self-contained project that I could own from start to finish was necessary.<\/p>\n

My engineering team, Production Systems, is responsible for the availability, scalability, security and costs of Intercom\u2019s infrastructure. From our backlog an intermission<\/a> stood out: reduce our Amazon Web Service (AWS) costs by up to 50% by maximizing our use of Spot Instances<\/a>. The cost of this spare AWS computing capacity varies on an hourly basis according to availability and demand.<\/p>\n

We needed to make the useof Spot Instances in Intercom safer and easier.<\/p>\n

We manage EC2 hosts with Auto Scaling Groups (ASGs). Amazon Web Services makes launching Spot Instances in your ASGs<\/a> quite easy; you just set the maximum spot price you are willing to pay. Sometimes the AWS Spot Market can send the spot price very high and clusters are left without capacity, making it dangerous to use Spot Instances for \u201cnear-realtime\u201d clusters. But if, during these market blips, we could replace Spot Instances with On-Demand Instances, where you only pay for EC2 instances you actually use, we could extend the use of Spot Instances, significantly reducing our AWS operating costs.<\/p>\n

At the beginning of the year, the team tried to deploy an open-source project called Autospotting<\/a>. From the description it seemed to address all our problems, but after deployment, we decided it wouldn’t work for us as it seems designed for less complex architectures than ours.<\/p>\n

We needed to make the use of Spot Instances in Intercom safer and easier and felt we had two options: fork Autospotting to make it work in our environment and get the changes upstreamed later; or build our own implementation of Autospotting with just the bits we needed.<\/p>\n

These options presented an interesting technical challenge, relatively small and with a big impact, exactly the type of project that would fit for my trip.<\/p>\n

Reducing friction<\/h2>\n

To optimize my two weeks away, I needed agreement from my team on a system design in advance in order to reduce communication friction while in another timezone.<\/p>\n

Autonomy doesn\u2019t just mean ‘working on your own’.<\/p>\n

Autonomy doesn’t just mean “working on your own”. It means you get to choose a project, own the design and set your own success criteria. At Intercom, we don’t require sign off \u2013 we are expected to do \u201csufficient due diligence\u201d ourselves and bring in other stakeholders and senior engineers as needed.<\/p>\n

This means early alignment on a design is critical to enabling engineers\u2019 autonomy. So I evaluated each option and shared my thoughts with the team for feedback. Here\u2019s a summary:<\/p>\n