Infrastructure at speed: 5 lessons learned from building Intercom in Europe

In December, we announced European data hosting, the result of one of Intercom’s biggest ever infrastructure projects. The lessons we learned while building out the infrastructure are invaluable as we continue to expand Intercom globally – as of April 2022, we are also hosting Intercom in Australia.

Until now, Intercom has been a multitenant application hosted in a single region in AWS. However, we’ve been talking to our customers and prospective customers about European data hosting for a long time – we knew what we had to deliver and the problem we had to solve: Intercom, but with the data stored and processed in Europe.

What we knew – and what we didn’t

We started out with a lot of “known knowns”; problems we knew we had to solve, such as multiregion software deployments. We identified some “known unknowns” too; problems we needed to solve, but didn’t know how just yet – like integrating the new region into our billing system. We were also sure there were plenty of “unknown unknowns” waiting to be discovered. These unknown unknowns made it difficult to estimate how long the project was going to take, or how many people we would need to dedicate to it. The scope was too wide to compare to other projects or work we’d taken on in the past, and the path to success unclear.

One thing we did early on was speak to teams at similar companies who had undertaken this kind of project before. In many cases, these projects had turned out to be some of the biggest these companies had ever taken on, taking the majority of their engineering team more than six months to complete.

“We were reluctant to slow down our R&D teams in the middle of a pandemic – so we built our project plan to reflect the way we like to work at Intercom”

Some companies went so far as to reimagine their architecture in the process. We were reluctant to make changes of that scale and slow down our R&D teams (in the middle of a pandemic!), so we built our project plan to reflect the way we like to work at Intercom.

That meant moving fast, despite the scale of the project. Moving fast while optimizing for the long term, as well as employing our principle of “ship fast, ship early, ship often” helped us not only to get the product off the ground, but ultimately deliver it to our customers sooner than we had planned for.

Lesson #1: Just start building – fast

Our dedication to moving fast brought us to our first lesson, and the decision that really unlocked the start of this project. In a recent Intercom on Product podcast, our co-founder Des talked about that old Jedi bell curve meme, and how it often applies to startups’ speed. Most startups journey through the “install more processes” stage until they finally realize they just need to move as fast as possible. Employing speed and hustle was going to help us to figure out those “unknown unknowns” and find solutions as we encountered them.

And so our former CTO and co-founder Ciaran Lee decided we were just going to start. We were going to begin building and go really, really fast, with a small ad-hoc team dedicated to the project – with the guidance that it was completely OK to fail.

“Being allowed to fail transformed our approach to the project”

If our approach didn’t work, we’d gain valuable learnings that would allow us to plan for something that might work in the future. In the best case scenario, we’d build something fast that worked well enough that we could start figuring out how to get it into our customers’ hands. Being allowed to fail transformed our approach to the project and enabled us to start moving forward. Instead of trying to anticipate problems and look ahead to guarantee success from the outset, we just started building until we hit a problem, and then figured out a solution.

It’s also important to note that we weren’t building prototypes that could later be used as building blocks for a later full implementation – we were building the real thing, figuring things out as we went along. The momentum we maintained as a result ended up being critical to the project’s success.

Lesson #2: Stick to your principles

Once we started building, our engineering principles helped us to keep moving fast. There were lots of ways we could go about building Intercom in Europe, including reinventing our architecture, but in keeping with our principle “Be technically conservative”, we chose to take the same approach we had used to build out the existing production environment.

“We didn’t just copy and paste, instead we shrank and simplified”

We introduced virtually no new software, services, or approaches into our Europe buildout. At the same time we drastically simplified our architecture, taking elements of our US infrastructure and reusing them in our new environment in a way that was much easier to work with. We didn’t just copy and paste, instead we shrank and simplified, aligning with our “Keep it simple” principle.

Lesson #3: Bend the rules when you need to

We needed to allow a lot of flexibility around our planning processes and team structures to get this project staffed and started, bending the “rules” while making sure to keep everyone informed about what we were doing. We built an ad-hoc project team including experienced engineers from existing teams to start working on the project.

Of course, there were consequences to this decision: teams were at lower capacity; project members had to straddle multiple daily standups; and other projects had to be deprioritized. This could never be our default approach for all projects, but when we knew what we needed to achieve, and we wanted to get started right away, it made sense to respectfully skirt our processes in favor of progress.

Lesson #4: Keep work as local as possible

This might be the most important decision we made to keep the project moving fast. Despite touching all parts of Intercom as part of the project, we decided not to farm work out across multiple teams, and instead kept as much work as possible local to our ad-hoc project team. As well as avoiding wider planning processes, it meant we didn’t have to ask our R&D teams to facilitate the deployment of our features into Europe. We avoided countless meetings, documents, and Slack messages by just doing the work ourselves as a default approach.

“We assumed ownership of the problem and empowered ourselves to make progress on it”

We assumed ownership of the problem and empowered ourselves to make progress on it, minimizing the overall cost to Intercom by minimizing disruptions to teams not working on Project Europe. On several occasions we did have to ask for help from people with expertise, and we caused a couple of surprises for some teams – but overall this was a massively successful approach.

Lesson #5: Keep timelines flexible

After we built out the infrastructure and ensured Intercom Europe was operational, we moved to a different phase of the project and worked with multiple teams across Intercom to coordinate the customer-facing launch.

Our launch blockers were largely our own internal processes, and several customer-facing integrations which we didn’t see as critical to the launch. So we asked ourselves, could we just launch without the likes of WhatsApp capability and fill in those gaps as we went? What was really holding us back?

“By looking at the timeline and assessing what was left to do, we figured we could pull the launch forward to December”

Our project plan had a January launch, but by looking at the timeline and assessing what was left to do, we figured we could pull it forward to December. We needed some help from customer support, sales, analytics, marketing, legal, R&D, and others, but everyone pulled together to move fast.

We have a Slack channel showing when our sales team closes deals with customers using the platform, bringing in real revenue for Intercom. The payoff of fighting for adoption in these last stages becomes clear in this channel – it compounds the value of all the work we put in to get this far. It would have been much easier to follow our existing timeline, but by pushing ourselves, we managed to get it into customers’ hands a month earlier than planned.

Our learnings will help us keep moving faster

This has been such an exciting project to work on – I’m proud of the work that we did, and that we minimized the project’s impact on teams across Intercom. There’s still lots of ongoing work, but the learnings we’ve taken from the experience were invaluable as we built out our Australian hosting and as go on to develop infrastructure in other jurisdictions.

Learn more about European data hosting with Intercom