Engineering principles: make it reversible

If your Facebook News Feed looks anything like mine, then it’s filled with short, snappy recipe videos.

You can watch them without sound; they’ve got subtitles. Before your eyes, using common ingredients you probably already have at home, something delicious comes together. I’m fascinated with these things, partially because I’m a terrible chef and they make it look so easy. I’ve always found following recipes to be inherently stressful.

Many of the steps in the recipe are irreversible. Consider stirring some eggs into a bowl of milk; there’s no amount of science or technology in the world today that can undo even that simple operation. I often find myself making a mistake along the way in a recipe, and then reaching a familiar decision point: do I go back to the very start, throwing away everything I’ve done so far, or do I carry on regardless, bravely hoping that everything will come together?

This blog post first began life as a talk on our Inside Intercom world tour. You can watch a recording of the talk above and the slides are embedded below. If you want to continue reading just scroll down.

Make it reversible – Stephen O’Brien at Inside Intercom Austin

from

Intercom

We’re familiar with this dilemma in the real world. We can’t go back in time. We have to be careful with the decisions we make and the actions we take.

There are some exceptions. Below is a cool video from a fluid mechanics lecture. The demonstrator here puts in three drops of color, he stirs them and we see them combine as we’d expect, into a sort of brown smudge. Then he starts stirring the other way around. Slowly but surely, a pretty remarkable thing happens; the three drops of ink reappear as if untouched.

Your browser does not support the video tag.

This is a truly reversible process. Looking at the two snapshots below, you can’t even tell which one came first in time. I wish that recipes were more like this, or the steps in recipes, such that you could go back and forth, make mistakes, and experiment. I bet it would be a lot more fun, and the food that I could make would be a lot better too. So I’m going to try and sell you on this concept of reversibility.

In our industry of software, we can do almost everything reversibly.

In our industry of making software products, at this point in time we have the luxury of being able to do almost everything we need to do reversibly. Let’s take a look at a few examples.

Engineering reversible code

As Director of Engineering on our growth team, reversible code is something that’s near and dear to my heart. It’s easy to forget, but until recently software was literally a physical product. It was finished in factories and it was shipped on actual ships.

The process looked something like this: We’d start with code. We’d write this maybe over a period of a year or more. And we’d do some quality assurance before physically manufacturing and shipping.

Those two final steps – manufacturing and shipping – were irreversible. When the code’s on the ship it can’t turn around. I can only imagine how stressful that would have been for large engineering teams, to effectively sign off on a massive set of changes and to say “They’re perfect!”.

Luckily, that’s no longer the case. Yet the factory mindset of the days when software was physical – the need to get it right the first time – has persisted. It’s an appealing and a dangerous concept. Today 70% of software companies still ship, at most, weekly. Many ship monthly or quarterly. The process has changed a little bit; we no longer physically produce things. We still batch up a huge quantity of code, a week, a month, or a quarter of code, and release it in a big bang to our customers. There’s still this irreversible step there at the end.

How we ship at Intercom

The way we do it at Intercom, and the way companies are increasingly doing this, is to continuously deploy. At Intercom any time an engineer makes a change – be it a bug fix, an improvement, or a brand new feature – we re-release our application. The process is like this: the new code will arrive onto GitHub, we’ll push it up to some automated tests that run in CircleCI or Codeship, we’ll deploy it using an internal system, and it’ll arrive on the cloud to our customers around the world. This happens in minutes. There are no red lines here; it’s a truly reversible system. If something goes wrong we can undo and that takes about a minute.

Engineers love this. They get addicted to it. Below is a graph of the number of times we’ve released per day over the last few years. You can see back in the early days it was just a couple times per day. Now we’re doing it perhaps 100 times per day. This is a truly continuous and reversible system. There’s a huge number of tiny changes streaming in, streaming out. Reversible code and continuous deployment, they get rid of the need for that factory mindset. We no longer need to be right first time. We no longer need to always be right. We can get away with being mostly right, just so long as we can quickly change our mind and fix things.

Applying reversibility to product

Your product development process essentially exists to answer these two questions: what should we build, and how should be build it? I don’t want to downplay the importance of the first one, but effectively you want to go from your company’s mission through your team strategy and work out what problems you should even try to solve.

We had an interesting issue related to this on our growth teams at Intercom. First of all, I’m going to back up to the initial incarnation of our team. We had Ben, our senior director of growth, whose job it was to conceive strategy and problems that our team would work on. Then you had myself, Patrick, and Hongyuan – engineers and a designer who would collaborate on solutions. We had all of our process there in that middle step, the solution phase. We had an irreversible step here, but we didn’t really realize it. We were a tiny, tight-knit team.

Fast forward about a year and a half, our team’s grown to more than 20 people. We’ve now got other folks like product managers whose job it is to care about strategy and about problems. We still only had process in that solution phase. What happened to us increasingly was that we’d be debating or critiquing a solution like a design, and we’d realize, “Oh no, we’re solving the wrong problem here.”

It’s a bit like that moment in a recipe when you make the mistake. You’re faced with that choice: do we go back to the start and abandon our work so far? In the case of code and design, do we press on and hope that this mistake won’t cause a problem down the line? In a few cases we had to abandon work we’d done, which was demoralizing and stressful for the whole team. We had to make this process reversible.

It took just a little tweak. We unbundled the problem stage, we added a few feedback cycles there such that we could propose a problem that wasn’t quite right, find out early and go backwards. Adding some process to review the problems, tools down, before we commit to solving them, made things reversible. This made things more fun, and it’s had a pretty big impact in just a couple of short months

To the second point of that product process; how do we build the damn thing? We can consider a feature as being sort of a stack of concerns, starting with concept at the bottom, working through design, and then systems and the UI that we need to implement for it to work. The tempting way to work through this pile is from the bottom up. Build the base of your pyramid and iron out your concept, generate some pixel perfect designs and throw them over the wall to engineering.

Again the problem is each of those little steps are themselves irreversible at times. You might find that when you’re instrumenting or you’re building your UI that you got some of your core design principles wrong and you need to go backwards. We recommend instead that you scope off a complete holistic slice that cuts through all of these concerns. Ideally, it’s the smallest set of changes that you could release on their own to customers. Perhaps it’s a beta version of a feature or a version of a feature which works in a restricted set of use cases.

Can you ‘cupcake’ it?

My favorite explanation of this comes from a blog post from Des. In this hypothetical example we want to make a wedding cake. The traditional, bottom-up approach here is to start with a cake base, make our filling afterwards, and eventually ice the whole thing. Again, those steps are irreversible, and we might find at the very end that because this is our first ever cake, it just doesn’t taste good. We have to start over.

Instead, we start with a cupcake. It’s the smallest, holistic version of what a cake could be. It’s something we can taste when it’s done and it exercises our whole kitchen. We always try and start with a cupcake at Intercom. You hear people in all sorts of teams in the company saying, “How can we cupcake this?” Meaning, how can we get a small representation of this out into the world to test our assumptions?

Sometimes you realize that a little cupcake is all you need – it does the job. And even when that’s not true, you learned a whole lot from it.

My advice to you is to think about your working processes, and look for any irreversible steps. The big splashes, the loud bangs. Ask “can we make this reversible?”. Often, with a bit of thought, you can. And I think it’ll lead to happier people and better results.

And remember this is a rule of thumb not an absolute. Sometimes, you want a loud bang. Sometimes we consciously choose to break this rule at Intercom. For example, when we introduce major new features, such as Smart Campaigns recently, we deliberately choose the stress of coordinating PR, marketing and brand all working together, because there are benefits to marketing and to customer education. Like all things, just be conscious of the trade off, and be deliberate about it.