Build boring software

We’re conditioned to think from an early age that exciting things are the best.

That attitude can extend to engineering, too. When trying to solve a problem with software, it can be tempting to build an exciting, complex solution. But that unnecessary complexity can lead to even more complicated issues. At our event Building Intercom in Dublin, I argued that when it comes to building software, we should avoid excitement and embrace boring. Watch the video above or read on for a lightly edited transcript.

When it comes to software, boring is best

A few years ago, I went back to college to study computer science. As a mature student, so I did all the things mature students do. I sat at the front, tried to answer every question and shushed all the kids behind me with their chap snats or whatever they call them. Even computer science students thought we were nerds.

It led me to have quite a big chip on my shoulder, so I always wanted to prove how smart I was and that I deserved to be there. So I made mistakes and tried to build really complex and exciting software, but it’s only recently that I’ve come around to the benefits of building boring software.

A while ago, we introduced a feature called Inbox reports, which gives you metrics about your support team, number of conversations, how quickly your team is getting back to them and how happy the customers are with those conversations. To create those metrics, we ran some big queries, we calculated the aggregations and then we cached the results. As our number of customers grew, the amount of data grew. We were putting more and more pressure on our databases.

“Like any dysfunctional relationship, it was exciting at the start”

To be honest, I like going home in the evening, and when I’m there, I don’t like being at the beck and call of a bit of software. This thing, however, was sending me angry late night emails and it also wanted all my attention during the day. It was like a dysfunctional relationship, and like any dysfunctional relationship, it was exciting at the start, right?

But it wore thin after a while, and I was exhausted. We decided to replace it with something more exciting for our customers, but more boring for us to own. I jumped straight in, wanting to show off as usual. I came up with this:

This was before any design or research was done. We have listeners passing data into a hand-rolled, artisanal ETL pipeline and loading data in through our fleet of Redshift clusters. I stood back and was like, “Yeah, I think I’m done here. I can’t think of any more ways to show off. This is done.”

But this is nuts. It would have taken ages to build, been a nightmare to own and cost us a ton of money to run every month. I was repeating the same mistakes I’d been making before. I wanted to set out and fix those mistakes that I was making.

“A boring problem is one that’s well understood”

It became clear that I was trying to solve a problem before I actually understood it, and a boring problem is one that’s well understood. You can’t expect to come up with a solution if you don’t understand the problem. Like those mature students I was talking about earlier, this solution was trying to answer questions nobody was asking. It was bloated and it came with a lot of baggage.

In order to understand the problem better, I had to realize that I’m not just an engineer. I have to be a PM and I need to be a designer as well. You need to get involved with the product decisions that were being made early on, so you can have an influence. And not just technically, because not many problems are purely technical.

Choose boring technologies

From there, I felt I understood the problem, so I thought I’d get back to the solution. Around this time, someone sent me a blog post by a guy called Dan McKinley. It’s called “Choose Boring Technology.”

In the post, he references former US secretary of defense Donald Rumsfeld – “There are known unknowns, and there are unknown unknowns.” McKinley talks about how unknown unknowns have a cost and converting them to known unknowns is quite time consuming and expensive. In the end, it can add unnecessary complexity to not only your team, but also your organization and your company.

If you think about adding a new technology to your stack, you should ask yourself these five questions:

1. How do we deploy it?
2. How do we maintain it?
3. How do we train people to use it?
4. How do we recover when it fails?
5. How do we develop with it locally?

If you have a technology already in your stack, you probably have really good answers for these already, because we don’t want to keep answering the same questions again. We use Elasticsearch quite a lot at Intercom, so I decided to ask the question, “Could I use existing tech that we have to solve the problem that I’m looking at?”

Compared with Redshift, Elasticsearch was much better suited for our use case. I didn’t have to answer all these questions again, and it knocked a lot of time off. I could just get on with building a solution. Now, with our boring Elasticsearch data store, we got rid of this fleet of Redshift clusters. We don’t need a load node anymore. We can just have a simple writer. We went with a much simpler event-based approach, so now we don’t need a whole ETL pipeline, and we end up with this simple architecture diagram.

Easy to understand, familiar and uneventful

Boring code, to me, is something that doesn’t confuse me. I like this quote by Brian Kernighan, “If you’re as clever as you can be when you write it, how will you ever debug it?” I’ll sum it up as,

“Being clever is dumb.”

I realize now I’m not as clever as I like to think I am, so everything I do is about trying not to confuse myself down the road. When you’re trying to solve a problem, you want to keep that capacity for solving the problem, not remembering a lot of stuff. Here are 3 tips on how to do that:

• Use patterns: We’re good at recognizing patterns. It means that you can push that down to the lower level stuff and you don’t have to remember what every other thing does.
• Be explicit: Remove levels of indirection, because it just means it’s less context that you need to remember.
• Be consistent: We have this concept of a signal, and everybody on our team, front end, back end, PM and design, all call it a signal. It just means my brain doesn’t have to map something to something else, and then onto something else. I can just get on with solving the problem.

What did boring get us in the end? We estimated about six months build time for the original design, and it ended up taking us just six weeks. It was going to cost us $70,000 a month to own, and now it’s just $5,000. We were able to get a lot more engineers on board and really speed up development, and most importantly, I got some rest, which was good.

“In my opinion, boring gets a bad rap”

Actually, the most important result was that our customers got this really slick and powerful feature that they had been craving for a long time.

In my opinion, boring gets a bad rap. When you think boring, you think about queuing, or maybe you think Ed Sheeran’s music. Even the definition of boring – “not interesting” – is boring. In software, I think the definition of boring should be: “easy to understand, familiar and uneventful.”

So let’s all get excited about boring. Thanks very much.

If you liked this post, read about our “Run less software” philosophy: