{"id":16804,"date":"2018-09-10T17:35:56","date_gmt":"2018-09-10T16:35:56","guid":{"rendered":"https:\/\/intercom.com\/blog\/?p=16804"},"modified":"2020-07-30T12:55:31","modified_gmt":"2020-07-30T11:55:31","slug":"product-engineers-balance-speed-and-safety","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/product-engineers-balance-speed-and-safety\/","title":{"rendered":"Break point: Product engineers and the pursuit of speed and safety"},"content":{"rendered":"

As engineers, one of the fundamental things we have to learn is the extent to which we can break things \u2013 how do we get the right balance between shipping safely and shipping fast?<\/p>\n

Before I joined Intercom, I thought of shipping as somebody’s else concern \u2013 I built things, but other people shipped them. It was a slow process, requiring a full team merging branches and deploying to production. Above all, it seemed risky \u2013 the potential to break everything loomed over the entire process. The process was so arduous, it would even end with a celebratory cake.<\/p>\n

You ship ambitious changes as a series of small, safe steps<\/p>\n

Almost 3 years in as a product engineer here, I have changed my perspective. It\u2019s almost impossible to work here and not learn how to ship safely. The core infrastructure that we run our code on was built with shipping in mind, supporting systems that need to change all the time. As a result, some level of safety is built-in, protecting us by design.<\/p>\n

Most new hires get super excited when they figure out that merging to master branch actually kicks off a new deployment. It\u2019s quite smooth. Nobody has to supervise, we are constantly shipping<\/a> (more than 200 times a day), with zero negative impact on production. If something goes terribly wrong, the deployment is reverted automatically. No cake.<\/p>\n

Small safe steps<\/h2>\n

Shipping with such a setup changes how you think about building software. You get in the mode of shipping ambitious changes as a series of small, safe steps<\/a>. Some changes are very straightforward (simple code change), some are moderately complex (adding a nullable column), and others are a little tricky and need to be deliberately split up and handled as a multi-step process.<\/p>\n

It can feel somewhat like the precision of surgery<\/p>\n

Between all the tools available and your peers’ experience, you can safely engage in a delicate, precise series of code and infrastructure changes \u2013 it can feel somewhat like the precision of surgery, except you know that the patient is not at risk.<\/p>\n

The first time I fully realized the effect of this was when I had to introduce a simple new functionality into the system. Could I be sure I would be able to turn it off if something went wrong? No problem, the mechanism was in place\u200a\u2014\u200aI just added a feature flag as a killswitch, wrote tests for both code branches and I was ready to roll safely once again.<\/p>\n

Some time went on, and I had to change the underlying datastore of Intercom’s Bulk API. Changing bit by bit, chopping bigger changes into smaller ones to lower the risk of the change, doing a series of surgical, safe maneuvers. This was a much bigger endeavor, but despite the scale of the changes I was implementing, the app did not go down.<\/p>\n

Shipping value faster<\/h2>\n

All these techniques help us ship all the time, in small increments, with no fear of breaking production. The process gets so ingrained in the culture that it becomes second nature.<\/p>\n

Can I trade some safety for speed?<\/p>\n

However, it can be easy to end up conflating two things \u2013 shipping the code safely and shipping business value. I find great comfort in doing these surgical maneuvers, but I\u2019ve recently started asking myself if there is a hidden cost to this approach.<\/p>\n

Above all, can I trade some safety for speed<\/a>? Is safety even important to what I am doing right now? These abstract questions helped me to uncover tradeoffs I had been doing subconsciously:<\/p>\n