{"id":15226,"date":"2018-04-27T14:05:35","date_gmt":"2018-04-27T13:05:35","guid":{"rendered":"http:\/\/blog.intercom.com\/?p=15226"},"modified":"2020-12-12T20:50:18","modified_gmt":"2020-12-12T20:50:18","slug":"rapid-response-how-we-developed-an-on-call-team-at-intercom","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/rapid-response-how-we-developed-an-on-call-team-at-intercom\/","title":{"rendered":"How we fixed our on call process to avoid engineer burnout"},"content":{"rendered":"

Intercom\u2019s mission is to make internet business personal. But it\u2019s impossible to be personal when your product is broken<\/a>. Uptime is critical to the success of our business, and not just because our customers are paying us, but also because we heavily dogfood<\/a> our own product. If our product is down, we acutely feel our customer\u2019s pain.<\/p>\n

Uptime is influenced by many factors such as the software architecture and the quality of day to day operations. However, quite often it comes down to having a human on call, responding to an alert from PagerDuty<\/a>. On call work like this can be a powerful customer orientated activity that connects engineers to the value customers get from your product. It can also be a great learning and growth opportunity \u2013 after all, outages and errors can be complex events to understand and remediate.<\/p>\n

Being on call out of office hours is inherently disruptive to your life<\/p>\n

But at the same time, being on call out of office hours is inherently disruptive to your life<\/a>. You need to be ready to respond quickly and competently to an alert about something being broken. Even without being paged, being on call creates anxiety \u2013 I know from personal experience that it is very disruptive to sleep, even if nothing actually breaks. Being on call regularly can lead to burnout, apathy or a general desire to never see a computer again.<\/p>\n

The history of on call at Intercom<\/h2>\n

Back in the early days of Intercom, our CTO Ciaran was the entirety of the on call team, both in and out of the office. As Intercom grew, we built an operations team to help Ciaran out. Soon after, new teams started building a lot of new features and services, and they assumed full on call responsibilities.<\/p>\n

There were too many people on call at any moment<\/p>\n

This felt natural at the time, as it was a lightweight way to scale our on call team and was consistent with our values that emphasized the importance of ownership<\/a>. Without deliberately planning it, we had ended up with four or five teams that were regularly being paged out of hours. The remaining product teams had a handful of alarms that rarely, if ever, resulted in an out of hours page.<\/p>\n

We realized that we had ended up with an on call setup we weren\u2019t proud of, and had a number of critical problems that we wanted to solve, such as:<\/p>\n