Having the right amount of process at your startup

Discussing software development and engineering processes is a great way to end unwanted conversations at parties.

Dull because it feels like a long-answered question. In the dark old days we used waterfall, now that we have iPhones and reality TV, we use agile. It could be an agile scrum, kanban or a process you’ve built yourself but the debate is over: everyone’s agile now.

These engineering processes are comforting, especially at a startup when it feels like you’re flying the plane while you’re trying to build it. Having any plan is better than no plan, right? Not really. Companies that insist on having a set process in place before they build product are doomed to fail because they’re starved of feedback for too long. Make sure your process isn’t sabotaging progress.

Where did it all go wrong?

If you’re lucky enough to be unfamiliar with the concept of development processes, they’re the way software teams figure out what needs to be built, how to prioritize it and who should do the work. It’s not a complicated idea, though there’s a whole industry of training and certification that would like to convince you it is. Waterfall is a “figure everything out before we start” model, loved by banks and insurance companies. Agile is more a “figure it out as we go” model, loved by tech companies and banks trying to look cool.

Sometimes we tightly grasp our agile artifacts and rituals but stubbornly insist on stepping back to the last century. The worst outcome? Building something we know is wrong because our process won’t let us stop. To better understand how we construct these traps for ourselves let’s look at where the old way, waterfall, came from.

Back in the 1950s people had seen small teams of competent engineers successfully deliver small- to medium-sized programs. Logically, if small teams make small programs then big teams make big programs. Round up a hundred people who can type, show them a computer and you’re ready to go! Naive but understandable in the first days of our industry.

Unsurprisingly, this way of thinking didn’t work so well. Herbert D. Benington, one of the contributors to SAGE – a US defense project that ran considerably over time and over budget, distilled the lessons of big projects into a paper: Production of Large Computer Programs. Page 11 has this diagram:

Herbert didn’t call this “waterfall” – that name wasn’t coined until the 1980s – but it’s exactly as we’d understand it. Figure out the problem, figure out how you’re going to solve it, design, build, test and ship. It’s a non-iterative flow of discrete steps drawn from the manufacturing world where the cost of change increases dramatically the later it comes in your process. As an industry we love to borrow from this world, taking ideas like Scrum and Kanban from auto manufacturers and photocopier makers.

When waterfall starts creeping into agile

Everything changed in 2001 when a set of principles were nailed to a pixelated photograph and the Agile Manifesto was born. The four core values are a huge mindset shift from the waterfall diagram above:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

Or, trust people over process.

Waterfall insists on certainty from start to finish. Know what you’re building, and how, before you begin. It makes perfect sense when you consider its manufacturing origins. If we built factories the way we build software only members of European monarchies could afford cars.

I’ve seen waterfall creep into agile teams when “handoffs” appear. Product hands off to design, design to engineering, engineering to QA. That same non-iterative flow of discrete steps. Sometimes the steps aren’t so well defined but the one-way flow remains, often disguised in agile language.

The only certainty is uncertainty

What characterizes both Agile and waterfall is a need for certainty. Certainty is re-assuring. It’s comforting. But it can also inhibit you. If there’s a plan, why doubt it?

What you really need is a process for building product that expects uncertainty. We write our plans before we start tackling a problem, but we’re mindful that this is when we know least about the problem we’re solving. Product, design and engineering need to experiment together, in small steps, to build a shared understanding of what the customer needs. Let prototypes, small iterations and as much customer feedback as you can get, guide you to the right path.

Be comfortable, and even happy, knowing that most of your early decisions and assumptions will be wrong. No product team can fully predict how their users will behave or react, so be skeptical of any process that locks you into your early decisions. I’ve worked on many projects where teams have agonized over tough decisions early on, only for them to be overturned soon after.

It’s easier to be mindful of this with design and engineering. As you build something new you have to try not paint yourself into a corner. You have to keep everything flexible to allow you to change your approach as you need. Organizationally, this is harder. The wrong goal can start you on the wrong path and the wrong process can stop you from changing direction.

Let’s say you’ve set a quarterly goal to build a new feature you’re certain every customer will love. You have announced at your all-hands and the whole team is on a mission to deliver. A month in and the first round of user testing shows you’re solving the wrong problem. There’s something far more useful in disregarding all your early work and taking another direction.

Does your process help you correct here, after you’ve learned something new? Or does it work to keep your momentum on the wrong path, the one you chose with much less information?

A good process isn’t about using the right buzzwords and rituals. A good process recognizes uncertainty and works to reduce it. It’s not enough to have an “agile” engineering team when the rest of the company expects you to work in a waterfall way. Everyone, CEO to first-day intern, should understand the primary goal of any project is to learn and that the most valuable learning happens through research, shipping and feedback.