Waterfall vs. Agile Development, an illustration from the NB DOT

This post is mostly about the Department of Transportation (specifically the one of New Brunswick), but first I want to thank the Department of Small Mercies for doing me a solid...or two...or three...

I was scheduled for a noon meeting in Moncton today.  The initial meeting announcement listed the location as happening on the campus of one of Moncton's two private one-year colleges.  Which narrows it down to a single street address, but nevertheless leaves something to be desired in terms of precision.

So, having enough sense of the organiser's temperament to allow myself a bit of smart-arsery, I enquired as to whether the vagueness was a test of my persistence/resourcefulness, upon which my admission to said meeting would depend.  It turns out that the organiser is at least my equal in smart-arsery.  Which I naturally took as a challenge:  "Oh, honey, it is soooo on," I thought, resolving to be in the room, greeting him with a wave and a Cheshire Cat grin, when he arrived.

Thus, I left Grande Digue with a little over half an hour to spare.  Outside of Shediac on (westbound) Highway 15, traffic suddenly slowed to the proverbial crawl.  First an ambulance and then a squad car passed our line on the left.  Somewhere past the 29km marker, a car lay flipped on its roof, but the rubber-necking was kept to a minimum.  Including my own, so my Gentle Reader will have to check the news for the details.  (P.S.:  Bonne chance, whoever you are...)

Following a short speed up, I encountered the stretch of road which had been stripped of asphalt during my previous run into HubCity.  Today the asphalt was fresh, a dotted line had been painted down the centre...and traffic again slowed down to < 10km/hour.  At least when it wasn't standing completely still.

And this is where our "illustration" really begins.  Because that freshly renewed stretch of highway had been necked down to one lane for kilometres.  Kilometres of perfectly sound road, missing only the side-line markers and (possibly) the rumble-strips.  With nary a worker nor piece of equipment in sight to justify the buffer-state of asphalt that kept traffic to the pace of a snail on quaaludes.  (And, as anyone who lives where other folks vacation can attest, there should be a Special Hell(TM) reserved for anyone who pulls those shenanigans during Tourist Season.)

In software development, there are two ways of making A Thing (for lack of a better term).

• The "Waterfall Development" school harks back to the assembly line of the industrial past (presumably an artefact of lumping Software Engineering in with traditional Engineering).  Designers hand off to Coders who hand off to Testers who ultimately hand off to whoever packages the code and delivers it to customers.  Henry Ford would feel completely at home in this world.
• The "Agile Development" school hews more to the "throw it against the wall and see if it sticks" line of thought.  Which, surprisingly, is also based on manufacturing principles developed in the automotive industry.  Except that it was done within the limited resources of a scrappy post-WWII Toyoda (now Toyota).

Obviously, both lines of thought have their place--Ford and Toyota are corporate behemoths decades after their founding.  "Waterfall" is particularly suited to leverage repeatable processes under economies of scale.  "Agile" thrives in uncertain markets, especially when the end-product might not even be fully conceived.   And, as with any diametrically-opposed methodologies, anyone who attempts to mix them does so under the cloud of impending disaster.  A Waterfall mobile app. start-up jumping the latest VC bandwagon is doomed; conversely, nuclear power plants are strangely averse to the measure-to-learn religion of Agile.  (Go figure.)

But that mixing of contexts is, alas, precisely what added an extra twenty minutes to a commute that normally takes thirty.

Context:  For my Gentle Readers outside of Canada, New Brunswick is what's known as a "have-not Province," and has been since steam replaced sail.  Historically, various Governments (Conservative and Liberal) have cushioned budget shortfalls with "equalisation payments" to guarantee a minimum standard of public services (notably those related to health care).  But New Brunswick's debt (and its debt to GDP ratio) has crept up under both brand-name parties.  And that's before the previous federal Government decided to jump on the austerity bandwagon.

What with that and the talk (a.k.a. threat) of privatising provincial road-building, it's hardly surprising that the bean-counters have taken over with a vengeance. 

Now.  Road construction is not my domain, we say in software development.  So some of the following will be, to a greater or lesser extent, conjecture.  That being said, I can in all fairness call out certain strong similarities between what I do for a living and how the folks in the bright orange jackets make their gelt:

• We only perform our ostensible "work" between interruptions.  In their case, it's mainly weather.  In mine, it's meetings and administrivia. 
• We can't always trust the infrastructure.  In their case it's a pocket of soggy clay, erosion from wonky grading on the last job, etc.  In my case it's network issues, unexpected upgrades, security holes, what-have-you.
• We can be screwed over twelve ways to Sunday by vendors.  'Nuff said.
• We can be--and too often are--encouraged by the Powers That Be to cut corners and/or kick the can down the proverbial road.
• We have to develop and learn to trust a healthy sense of pessimism to sniff out the edge-cases that could bring everything crashing down.
• We know that nothing is ever going to be 100% perfect 100% of the time -- there will always be "beta" mode as well as maintenance.  More on that later.

And, thus, I don't think I'm going out too far on a limb to say that road construction is--or at least should be--more of an Agile enterprise than a Waterfall enterprise.  A road that holds up under years of traffic is most certainly not made by a paper-pusher in Fredericton.  It's made by the experience and the up-close-and-personal first-hand observation of people whose summers are spent with sinuses full of dust and tar.  Not unlike how a project manager can't necessarily appreciate how many ad-hoc hacks will have to be added to deal with the gawdawful third-party legacy system that the designers never bothered to check before the contract was signed.  [involuntary twitch]

I mentioned "beta" mode above.  A road that's partially open during construction is basically the same thing as an open beta in software development.  And in open beta you emphatically do not wait until the "official" launch-date to release all the bug-fixes and missing features.  Any product beta-tested in that fashion will lose the interest of the influential early adopters (and probably never see the light of day).  No.  You shove those things out the door as soon as QA green-lights them.  (Granted, the province has the upper hand in this instance, because people will always gravitate back to the shortest time between Points A and B.   But my point, I think, stands.)

Yet, whoever was directing today's crew could have easily limited the bottleneck a mere kilometre of work surface and left everything else open to two lanes of traffic.  When that was finished, they could have done exactly the same thing for the next kilometre of surface.  And so on...until either the potholes or budget ran out.  In short--the Agile method would have produced the minimal amount of traffic disruption during the busiest season of the year.

Instead, someone (quite wrongly) chose to operate by the Waterfall method.  Again, this is just conjecture, but my strong suspicion is that that someone assumes that handling the resurfacing in larger chunks allows for economies of scale.  For all I know, they're correct--at least superficially.  And, of course, the Suits luuuuuuvvve their Waterfalls...mainly because they live upstream and it makes them feel more like they're the driving force behind everything.  [eyeroll]

But the problem is that, by my estimate, the entirely gratuitous one-lane bottlenecking cost each and every person about extra fifteen minutes, compared to the length of road actually being worked on.  Every missed appointment, every late delivery, every disgruntled tourist --  those come with a cost, too.  For sure, that cost won't show up on this year's tax bill.  But it will show up in one way or another--make no mistake about that. 

In my particular case, the Patron Saints of traffic lights and 2-hour parking and sheer dumb luck (plus my taste for harmless practical jokes) allowed me to make my meeting just in time.  That, however, doesn't mean that I'm not brassed-off.  I've been on well-run projects and ones that barely ran at all--in both Waterfall mode as well as Agile.  Like I said, they each have their proper context.  And it is a sorry manager (and steward of public funds) who chooses the wrong context.