Run less software

It’s hard to win a battle you don’t realise you’re in. It’s even harder if you don’t know all of the armies on the field, their strategies and weapons, or even who’s a friend and who’s a foe.

The same is true in software. We are all in a battle, multiple battles in fact, with a lot at stake: whether it’s the fate of the company we work for or for the product that we build.

In this battle, I’ve found a secret weapon hidden within one of our core engineering strategies, an idea called Run Less Software. As well as being a critical philosophy behind how we build software, it also represents how I feel about the software industry and technology in general.

To understand why Run Less Software is so important, let’s map out the battlefield, the armies on it, their missions and their strategies.

Innovative startups

The first army are innovative startups. In simple terms, in order for a company to succeed they must have a good business idea and be able to consistently execute and evolve that idea better than anyone else.

When I say “execute,” I don’t simply mean the engineering challenges of building something. I’m referring to the full spectrum of business execution, from product management to design to engineering to marketing to sales to support and all the other functions needed to scale a business.

Copycats

The second army are your main competitors. They have the same great ideas and are trying to beat you at your own game.

In today’s world it has never been easier for your competitors to achieve this. The traditional barriers to entry are evaporating. Money is cheap. Interest rates are at all time lows. Investors are incentivized to put their money anywhere other than a bank. If your ideas and founders are credible then you can probably raise money relatively easily.

Basic technology execution is becoming easier too. The arrival of cloud computing, IaaS, PaaS etc. means people can fulfill almost all of their technology infrastructure and platform needs in minutes, cheaply, with only a credit card and an email address. Frameworks like Ruby on Rails, Node, Angular, Ember, React etc. are making it easier and faster for software developers to develop complex software applications atop this infrastructure. What took you months to ideate, design and develop can now be copied in days or weeks thanks to these new tools.

Basic marketing and selling at a global scale is becoming easy too. The Apple and Google Play app stores give developers an instant marketplace, a gateway to billions of customers. Facebook and Google provide an easy to use, high precision advertising engine that lets you target any cohort of people in the world.

If you’re still unsure about whether it’s easier than ever for your competitors to beat you at your own game, just look at three real examples from recent history – Snapchat (under attack from Instagram), Blue Apron (under attack from Amazon) and Hipchat (under attack from Slack).

The talent

The third group are the talented product builders all around us. We are the soldiers companies are trying to recruit to help them win the battle for talent. Demand outstrips supply 3:1. On the one hand, you might think this is a good thing. There are more technology jobs than ever before. But as technology advances and provides us with more advanced software components it devalues and commoditizes what we’re working on and at worst makes us redundant.

The Four Horsemen

The final army is led by The Four Horsemen: Amazon, Google, Facebook and Apple, the four most influential and dominant technology companies on the planet. (In past lives I’ve been a soldier for two of The Four, Amazon and Facebook). Whether it’s selling ads or storing data, The Four have become the nerve center of businesses all across the world and therefore take a cut from almost every business, regardless of whether it’s winning or losing its own particular battle for market share.

Now we’ve mapped out the battlefield, the armies, their intents, weapons and strategies, how should your company react? How can your business win market dominance and survive? And what the hell has this all got to do with Run Less Software?

In The Matrix, Neo is able to defeat his most powerful enemies by wielding time. He’s able to slow time down and perceive it more quickly than his competitors, which gives him more time to react and defeat his foes.

We don’t have the same powers as Neo does, but we can learn much from his general strategy. We can take steps to help us see, understand, decide and execute more quickly and effectively than our competitors. This is the essence of what the philosophy of Run Less Software is all about.

There are three pillars of the Run Less Software philosophy, all of which will help you manipulate time and defeat your enemies.

1. Choose standard technology
2. Outsource undifferentiated heavy lifting
3. Create enduring competitive advantage

Let’s explore these in some more detail.

Choose standard technology

This is a philosophy worthy of its own presentation. In fact it already has one, put forward by the awesome Dan McKinley from his time at Etsy. I highly encourage you to read his blog post and watch his talk.

The philosophy is all about choosing “boring” technologies and patterns instead of using new ones.

Let’s say I’m a soldier in medieval times. In theory, I can choose 1 of 5 swords. It would be incredibly foolish of me to bring all 5 of those swords into battle with me. You can’t use one sword for your tall enemies, and another for your smaller ones. You just need a reliable sword that will defeat all of your enemies regardless, and make sure that everyone in your army is an expert in it.

Choosing standard technologies is very similar to this, only in software. You need to constrain yourself to solving problems mostly, but not exclusively, with a small, specific set of standard tools. By doing this over time we become experts in them. Then, we are able to build better and faster solutions from them.

I summarize it by using the following formula:

It means the best way to evaluate any engineering decision is to weigh up the operational costs and risks arising from it against the impact it’s had on your velocity.

When we constrain ourselves to the smallest practical set of technology building blocks and use these building blocks over and over again we become experts in them. They become easy and cheap to operate and fast and powerful to use. By choosing standard technology we systematically increase our speed and reduce the cost of our engineering decisions.

Like all principles, choosing standard technologies is a starting point, not dogma. Exceptions are fine, but they should be few and well considered. Do introduce a new technology when you and your colleagues have talked it out in detail and still believe the benefits outweigh the costs.

Outsource undifferentiated heavy lifting

In 1963 Peter Drucker published his article, Managing for Business Effectiveness, where he said:

This means finding an easier way of doing things, avoiding complexity and dropping non-essential tasks.

Almost 40 years later, Jeff Bezos echoed Drucker’s thoughts:

By undifferentiated heavy lifting (UHL), Bezos was referring to things like figuring out which servers to buy, how many of them to buy and what timeline to buy them. This work is necessary and important but it doesn’t help you win against the competition.

To do that, you have to make sure the majority of your energy, time, and dollars goes into your core idea.

Since 2006 the era of “Everything as a service” has advanced quickly. Cloud computing is dominant. Infrastructure as a service is now standard, as are Platform and Software as a service. Function (also known as Serverless) is about to be the next big thing. AI is potentially the next elephant waiting to enter the room. There are more opportunities than ever to outsource your undifferentiated heavy lifting to third parties.

These third parties are relevant because our engineering capacity has never been more limited. Thanks to the constraints of time and money and the scarcity of engineering talent, we must be very careful in deciding how to spend our precious time. We should be spending it on things that maximize value and minimize cost, both in the short and long term.

There will always be some unavoidable amount of cost incurred whenever your code runs in production. You will have to fix bugs that impact your customers, handle outages, etc. These costs can never be completely eliminated but what you can do is minimize them.

Offloading as much responsibility as possible to third parties is a useful technique for minimizing the number of things the team is directly responsible for (and therefore need expertise in) and allowing the team to operate at a higher level of abstraction (and therefore deliver customer value more rapidly).

Knowing what to outsource and to whom is an art, so we use the following guidelines to help us make decisions:

• Don’t outsource things core to your customers’ user experience as this makes it easier for others to copy your product and disrupt your business.
• Do outsource things the industry considers undifferentiated heavy lifting and enable your team to move faster, leaner and provide a higher level of quality to customers.
• Be extremely mindful and diligent in understanding the security of companies you are outsourcing to.
• Understand the potential consequences of outsourcing to young startups. Young startups go out of business regularly and their security and reliability is often worse than more mature companies.

Accordingly, we have a tiered set of recommendations for who to outsource to:

Over the years we’ve doubled down on the idea of outsourcing undifferentiated heavy lifting and have reached the point where close to 60% of our undifferentiated heavy lifting has been successfully outsourced. I like to think Jeff Bezos would be proud.

Create enduring competitive advantage

Tyler Durden from Fight Club once said, “The things you own end up owning you.”

The same principles apply in business. The things you spend most of your time on end up owning you. A valuable question for any software company to ask themselves is: “What do you spend most of your time doing and are you happy with it owning you?”

At Intercom, we build a SaaS solution for our customers, but we don’t consider ourselves a software company. We are a product company. The people who spend their days writing software are not called software engineers. They are called product engineers. We might have trained for years in college to learn how to write software (and this is a sharp tool in our tool chest) but our primary job is to solve problems for customers. We want to spend as much of our time working on the hardest and most valuable problems our customers need solving, not on reinventing the technology wheel.

To help us do this well, our engineers spend a LOT of time collaborating in a multidisciplinary product team. This means helping our research team conduct user interviews, participating in ideation sessions with a product manager and the product leadership team to evolve our roadmap, attending design review meetings to help them better understand how we are trying to solve a particular problem, or partnering with an analyst to better understand how a particular feature is being used by customers. We believe great product and enduring competitive advantage is created at the intersection of product management, engineering and design, and we work hard to create as much overlap of these three disciplines as possible.

Of course, our engineers are highly skilled at creating software and do still write plenty of code. But their work is on things that are absolutely essential to our company and to building a competitive advantage.

Broadly their work falls into three categories:

• Building new customer-facing product that solves real problems and creates real value for our customers.
• Evolving our product to make it better, safer, faster, cheaper and able to function at bigger and bigger scale.
• Investing in things that enable us to move faster and leaner for the long term.

Our engineers build and evolve our customer-facing web and mobile applications’ UI and workflow engine and our end-user facing messenger. Without these, Intercom wouldn’t be where we are today. They are absolutely critical to our business and are therefore high-value things for us to work on.

Our engineers also write a lot of Ruby code, which knits together a lot of AWS, infrastructure, platform and SaaS technologies that form the core of Intercom’s backend infrastructure. We’ve outsourced as much as we can, but we do still build and operate some backend technologies ourselves, like the backend for Intercom’s messenger. Considering the messenger is fundamental to our business, we believe we need to fully own and control it.

Finally, we own and operate much of our software deployment stack. In theory, this is undifferentiated heavy lifting. However, we deploy new code at a pace that others deem crazy or reckless (we ship new versions of Intercom almost 100 times per day).

This level of speed and agility is crucial to our success and we are unapologetic and unwavering in this belief. We’ve not yet found a suitable third-party provider who can meet our needs here and so continue to invest in doing it mostly ourselves.

In short, the things where we’re spending our time on are worth it.

Now that we’ve been through the three pillars of the Run Less Software philosophy, let’s review some of the technology battles where we’ve deployed these pillars and see, in the cold light of day, whether we won or lost and what we learned.

You need to constrain yourself to solving problems mostly, but not exclusively, with a small, specific set of standard tools.

The simple case

Sometimes the principles of Run Less Software are straightforward. You standardize a technology choice and outsource it to a world-class third-party provider.

For example, about three years ago we were running multiple different types of relational databases (some PostGres and a lot of MySQL). Administering two database technologies gave our engineering team an incredible amount of cognitive and operational overhead (time that could be better spent elsewhere) so we decided to standardize on one database technology.

We briefly considered running our own databases as they would have provided more power and flexibility, but at a higher human engineer cost. We quickly discounted this as it didn’t fit with our three pillars of Run Less Software and decided to consolidate on MySQL, specifically on AWS MySQL. As we were able to benefit from the pace of AWS innovation (and adopt AWS Aurora when it came along) we got 5x throughput at a 30% cost reduction. By reducing the amount of services we relied on, we reduced the amount of time (mental and physical) required to operate the new system.

The not so simple case

In other cases, the philosophy of Run Less Software is more difficult. It involves deconstructing a complex problem into smaller pieces and solving it by knitting standard technology building blocks together.

About 18 months ago, our primary user data store was running out of capacity and starting to creak. The system was based on MongoDB, a wonderfully flexible database technology but one that is hard to scale and not natively supported by AWS or any third-party that could meet our operational needs.

One option was to double down on MongoDB, hire some experts and really commit to running this software ourselves. However, this approach ran against our Run Less Software philosophy, so we needed to find a more efficient way to scale our database technologies.

The art of choosing a standard technology is that when you encounter a new problem, you need to break it down into a series of subproblems that can be solved using your standard patterns and tools. Following this approach, and using the lessons we’d learned trying to model our user storage needs on individual technologies, we were able to break down the problem into two separate parts which could be individually be solved using AWS Aurora and AWS DynamoDB. All we had to do was write a small amount of highly performant Ruby and Go code to knit the two standard technologies together. It meant we were able to solve a huge scalability problem much more easily than ever before.

It’s worth noting that arriving at this elegant solution took about two months of prototyping and many discussions among our smartest people. Even arriving at a “simple” solution requires great skill and creativity.

The world of software has never been more competitive. Technology is spitting out innovation after innovation at an unprecedented rate. This exponential rate of change in software means the temptation for many people is to throw themselves into the latest technologies so they don’t get replaced by an AI function. But as engineers and product people, we must be very careful in deciding how to spend our precious time. We want to spend it doing things that maximize delivery of value and minimize cost, rather than chasing the next big thing.

By following the three pillars of Run Less Software – choosing boring technology, outsourcing undifferentiated heavy lifting, and create enduring competitive advantage – we can move fast, solve real customer problems, and stay focused on the things that really matter to customers, while bringing reward and revenue to those of us building software. Get it right, and you’ll be best placed to defend yourself in any battles on the horizon.

If this sounds like the sort of engineering team that you’d like to work on, we’re hiring.