Page 1 of 2 in the bdufwaterfall category Next Page
 Earlier today, while reading a posting on an internal Microsoft DL dedicated to agile shop talk, I found a link to an Engineering Excellence wiki dedicated to Scrum. Whoa! It appears that my favourite agile/lean/iterative process has roots going back at least to 2004 within the company and that folks were seriously discussing its merits as a delivery model.
One section of the wiki dealing with the Theory of Constraints as applied within Scrum immediately caught my attention: Within it was a link to a page detailing The Five Orders of Ignorance. This sounded very Zen to me. It also conjured up images of kung-fu warriors trying to attain enlightenment from their master as they ascended the steps of their craft.
But that’s just me.
In reality, the Five Orders describe how much we do or do not know about a problem, with the understanding that different processes are more or less appropriate fro addressing problems with differing degrees of uncertainty. At first glance, this seems quite in simpatico with Boehm/McConnell’s observations with the Cone of Uncertainty around problem domains and when it is most appropriate to estimate a task.
The Five Orders of Ignorance were written eight years ago by Philip G. Armour in an article he wrote for The Communications of the ACM (Association for Computing Machinery) by the same name. In it, he defines them as:
The 0th Order: Lack of Ignorance
- When you know something and can demonstrate your lack of ignorance through some tangible form.
The 1st Order: Lack of Knowledge
- When you do not know something and can readily identify the fact.
The 2nd Order: Lack of Awareness
- When you do not know that you do not know something. In other words, you’re not only ignorant of something, but you are also unaware of the fact.
The 3rd Order: Lack of Process
- When you do not know a suitably efficient way to discover that you are unaware of something.
The 4th Order: Meta Ignorance
- When you do not know about the Five Orders of Ignorance. This is very Douglas Adams, as having just read this bullet, you now know that there are Five Orders of Ignorance. Don’t panic.
So now I know what I should know, but didn’t know and in some cases can’t know since I didn’t know I didn’t know in the first place. Now what?
Using the Five Orders, we can begin to classify the unknowns and uncertainties and how we manage them within the context of a project. A few years back, the press and late night personalities had a hay-day with a comment that Donald Rumsfeld made with respect to the activities of insurgents in Iraq and Afghanistan:
Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.
After hearing the preceding sentences, the last one sounded positively hilarious: How on Earth could there be unknown unknowns? Bwa-hahahaha. However, in reality this is just an expression of the 2nd Order of Ignorance. Note that this doesn’t mean “ignorant” in the pejorative sense – this is about the recognition that more information is needed and that we’re operating within the context of low-knowledge.
Gee. Sounds an awful lot like most software/IT projects, doesn’t it?
The anti-pattern to 2OI is when we choose to do nothing about it, which takes us to the 3rd Order of Ignorance – this is probably best expressed in BDUF/waterfall projects where we make an explicit statement about not knowing the unknowns and really have no plan for addressing them because we actively deny the existence of unknown unknowables through rigid project phases. The project plan is the plan. And it is all-knowing. The plan is the Holiest of artifacts and must never be questioned, etc., etc., etc.
But it gets worse: If an unknown unknowable should legitimately come our way and intrude on our carefully-crafted MS Project Plan, we treat it like a bugblatter beast of Traal and promptly wrap a towel around our head in the belief that if we can’t see it, it can’t see us.
And so the madness goes ever on – the condition of low-knowledge persists, paradoxically in the face of the very things that could dispel the “ignorance” and take us closer down the ladder to the 0th Order.
The Five Orders of Ignorance provide us with an indication of how we address wicked projects and their inherent problems: Knowing that we can’t know everything about a problem domain and that we need an adequate process to manage this “ignorance” is the key to successful delivery. This is a fundamental underpinning for all agile/lean/iterative processes: Ignorance isn’t viewed as a contractual failing (ie. “what do you mean schedule/cost is slipping because you didn’t realize ‘x’ – you should have known ‘x’ !) but rather as an expected opportunity for gathering new knowledge about the domain.
Via today’s Ottawa Citizen, this article: Aurora update soars $132M over budget: Aircraft upgrades began before DND knew extent of project
The CP-140 Aurora is a reconnaissance aircraft that has been used for over twenty seven years by the Canadian Department of Defense to patrol our borders – for the past nine years they’ve been undergoing retrofits to update sensor arrays, avionics and computers which, to the shock and dismay of a Government auditor, have gone $132M over budget:
The program to upgrade the CP140 Aurora's computers and avionics began in 2002 and was supposed to cost $197 million. But that figure has ballooned to a little more than $329 million, according to a newly released Defence Department audit of the project.
The audit, completed in August, found that the modernization program was difficult to manage because the Defence Department didn't know the full requirements of the project until it was well under way.
Of course they couldn’t know that! Who could? The author and auditor seem to suggest that there is a great human failing when contractors’ powers of clairvoyance fail them! As we get further in, the full spectre of the project’s failure is revealed:
"Since the CP140 modernization program was incremental in nature, the full requirements were not known until year four of the nine-year contract, resulting in amendments that made it difficult to manage and invoke the appropriate terms of the contract," the audit conducted by the department's Chief of Review Services states.
I’m not so sure about what’s meant above about the connection between an “incremental in nature” program leading to feature/scope creep that blew out BDUF estimates – I’ll have to assume that we’re looking at a series of waterfall projects, each compounding the problems of the other.
In any event, this type of fail-late exercise should be immediately familiar to most folks in IT/software development. It’s what happens in 90% of your projects.
By June 2005, the amendments to the contract, which included the need for training simulators and the integration of additional sensors, boosted the contract's value 67 per cent over the original estimate, the report noted.
Because the audit on the Aurora data management system contract is so heavily censored, it is difficult to determine whether the costs are still climbing. The project is to finish in 2009.
Again, if you’ve been on more than a few enterprise scale projects this will be immediately familiar to you: The “churning” that results from gathering new requirements, implementing them a la waterfall and finding out that they’ve introduced bugs requiring more gather-code-fix cycles.
Moving on, here’s where the matter gets a little audacious vis-a-vis the audit:
The audit was not designed to assess the performance of the companies involved, but was focused on the department's processes and practices. The names of the firms working on the Aurora data management system contract have been censored from the document.
Why not assess the performance of the companies involved? Are they failing because of their own flawed practices or is it a result of DND’s project requirements, which I suspect bear more than a passing resemblance to the U.S. DoD STD 2167 project stipulations from over 36 years ago – for the uninitiated, they specify all projects to be executed in a single-pass waterfall process.
Unsurprisingly, the auditor comes to the conclusion that it’s nearly impossible to assess true ROI (return on investment) – but not for reasons of the process, but rather that the amendments weren’t put out for bid!
The audit notes that some of the amendments to the Aurora contract were awarded without competition. "Value for money assurance cannot be provided for amendments worth (censored) because they were sole-sourced to the prime contractor," it added.
Balderdash. If you farmed these amendments out to other BDUF/waterfall contractors, you’d be just as likely to have even more compounded problems as teams who are unfamiliar with the project need to ramp up as part of their execution. The problem is not with sole-sourcing. It’s with the process employed. ROI cannot be realized when projects are integrated and tested so late in the delivery cycle.
I find it both amusing and frustrating (in the manner of continually hitting one’s thumb with a hammer) to see failed BDUF/waterfall thinking so pervasive – some try to duff it off as “the confluence of many factors” contributing to failure. True, but why not then remove the #1 factor that almost assuredly guarantees failure: BDUF/waterfall?
Ed.: I've made some minor changes to this post to improve flow and revise the conclusion.
A few weeks ago (October 16 to be precise…) I happened across an intriguing blog post by developer David Christiansen entitled Is there REALLY any rigor in Waterfall? Far from being a standard pro-agile polemic, Christiansen describes his own intellectual journey from a self-professed waterfall-neutral “fence-sitter” to agile/iterative-friendly convert through firsthand experience and research. Rather than accept prima facie the rhetoric on the problems with phased delivery, he researched the model’s pedigree and coupled it with his own experience and had his own revelation:
“For the last couple of years I’ve sort of ridden the fence on where I stand on project management. I’ve conceded that waterfall might have a place in the IT world in the past, that it’s a tool we should have in our toolbox, even if we don’t use it very often…
The waterfall approach to managing software development projects is one of the most costly mis-diagnoses ever. It structures projects for failure. It creates a cost-change curve that is unacceptably expensive (you know, the cost of a change grows exponentially as the project progresses) and is largely responsible for the high failure rate of IT projects. Waterfall creates a project structure that is rigid and forces you to make decisions and predictions when you know the least, then abide by them no matter what realities you expose in the process.
I am off the fence. Waterfall doesn’t belong in any IT project manager’s toolbox. It is based on and littered with myth.”
While I do enjoy reading about another developer’s Conversion on the Way to Damascus, David’s piece stands out to me because outside of Craig Larman, he’s the only person I’ve come across who makes reference of Winston W. Royce’s role in the story of the waterfall model. While Royce isn’t really responsible for the waterfall model – that came about as a result of our industry’s 30 year failure to RTFM – knowing who he is and the history behind phased delivery is vitally important to putting the rationale for agile/lean/iterative models into their proper context.
Is there a One True Way?
A week later, David’s post was picked up in a feedback thread to a rather pedantic advice column on InfoWorld that was responding to a student’s query on the value of requirements vs. testing (btw, I think both the instructor and the student are wrong) with the ensuing fracas prompting David to write a follow-up post where he expresses some disbelief about the fervor that seems to pervade the pro/con waterfall debate, ie. that there’s only “one true way” to deliver software: (emphasis mine)
Frankly, I’m perplexed by the belief among many technology practitioners in the ideal, the “one true process,” that will solve any problem, whether it is Waterfall, Spiral, Agile, or whatever comes next. It is a fixation I believe is unique to IT - I don’t remember seeing this type of zealotry when I worked as a manufacturing R&D engineer at Bell Helicopter. Engineers there had a much more experiment, do-what-it-takes sort of attitude that couldn’t be captured, analyzed, and then reduced to a simple prescription for project success.
With all due respect (and a wink and a nod), David’s probably not thinking of some of the holy wars that are waged in other disciplines like physics, quantum mechanics, climatological science and medicine when it comes to “one true way” zealotry. Hell, look at the trouble Robin Warren and Barry Marshall had to overcome in gaining acceptance of their evidence that a bacterium and not spicy foods were responsible for the vast majority of gastric and peptic ulcers. Intransigence and zealotry can be found just about anywhere.
But I digress: I also have to disagree with David's observation that the "experiment, do-what-it-takes" attitude of the Bell R&D engineers can't be captured and turned into a prescription for better project outcomes. In fact, this is the very essence of the models used in manufacturing (ie Toyota Production System, Lean) that agilists have adapted into a variety of methodologies from Scrum to XP to Crystal Clear to Lean.
Thus, while I agree that there isn’t an out-of-the-box “silver bullet” that will make every project successful, I am of the opinion that practices which belong to the family of empirical process controls have the highest probability for successfully delivering working software than any other process because they have significantly better visibility into project activities and an internal feedback-loop to support self-correction based on the realities of the situation at hand rather than a six-month-old plan.
Agile Values
David concludes his post by observing that in the final analysis it may be better to rely on human characteristics such as “brilliance, bravery, strength, and intelligence” to solve our problems than a naive and idealistic belief in a singular “right” way to build software. I think David’s got this half right, because I think he’s not fully aware of the values that back-up agile/lean/iterative practices – and I’ll grant that the zealots probably don’t know them, either.
For example, take the first value statement of the Agile Manifesto:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools.
Next, consider the Four Values that Kent Beck defines for project success under eXtremeProgramming practices:
We will be successful when we have a style that celebrates a consistent set of values that serve both human and commercial needs: communication, simplicity, feedback and courage.
Finally, consider Scrum creator Ken Schwaber’s comments on the challenges and rewards for implementing agile practices:
Of all organizations that attempt to become a world class software organization (by adopting agile practices like Scrum) only one third will make it.
The inspect and adapt practices of Scrum is where the pain comes in, because when you start doing that you discover everything in your organization that’s not working.
Agile is hard work; it’s a long process and it is extraordinarily satisfying
These aren't the words of idealistic zealots, but reasoned pragmatists, and they demonstrate a closer affinity to the core "human" values that David seems to cherish. While he may be dissuaded or alienated by the zealots who proclaim "one true way", the fact of the matter is that agile/lean/iterative disciplines (and that is what they are) work because they are built upon the fundamental understanding that building software is about people and how they can work best in a very complex and creative enterprise.
Earlier today while sorting through some digg links, an article on Tech Republic caught my attention: Understanding the Pros and Cons of the Waterfall Model of Software Development. By title alone, my spidey senses were tingling. No good can come of this.
First off, the author does state that this is a “quick and dirty introduction to the model” - more dirty than anything else, but ok.
I won't recount the entire article here since you can link to it; instead, I want to focus in this post on what the author (one “Contributor Melonfire” -- oh, that's rich) perceives as the pros and cons: Waterfall Pros:
- Enforces discipline - every phase has a defined “start“ and “end“ point allowing for progress to be conclusively identified through the use of milestones;
- Emphasis on requirements and design before writing a single line of code ensures minimal wastage of time and effort an reduces the risk of schedule slippage and customer expectations not being met;
- Quality is improved as a result;
- Aids in effecient knowledge transfer (via a specification doc) when team members are geographically dispersed.
Waterfall Cons:
Now, to be fair, the cons the author lists seem to be cribbed from some light reading of agile processes - fair enough, though I'd have preferred seeing more scrutiny.
- Inefficient in real-world situations because features and requirements are emergent over the lifetime of the project;
- Estimation is difficult due to up-front requirements capture;
- Recommended only in situations where requirements are stable and known;
- Assumes an ideal dev team model where tasks can be divided into segregated boundaries.
If the intent of this article is to provide the unitiated with an “honest broker's” assement of the waterfall model, it does a rather poor job. The “Pros” are almost laughable and demonstrate that either the author has no experience trying to deliver software this way or that he/she is a relative newcomer to the game, yet to be scarred like the rest of us.
Let's assume the latter and take them behind the proverbial woodshed for some schoolin'
First off, this notion of “enforcement of discipline” is entirely without merit. Waterfall provides the illusion of discipline and progress because it assumes that “the plan” (as devised by some sage wizard) is infallible, practical and achievable and through the sprinkling of some magic milestones will actually demonstrate and provide value to the customer. It does neither and falls into one of my own Seven Deadly Sins of Programming: “Delivering what the customer asked for, not what they want“.
Reality Check on Aisle 3:
This just doesn't happen; in the real world, the milestones get blown through like a runaway train through crossings -- because the plan and the estimates were made in abstentia of the team who has to deliver on them, they're next to worthless and convey a false sense of security. In addition, they don't provide the feedback loops (ie Sprint Demos) that help guide and steer an agile project through determining if what was planned is actually what the customer wants. It has but one conclusion: A shocked, dismayed customer who proclaims that this isn't what they're going to pay for.
With respect to the second “Pro” that Waterfall ensures minimal waste of time and effort because of the up-front investment in design and mitigates schedule slippages and customer disappointment: Um, how? Magic?
Again, the author's naivite comes through: Waterfall is all about making commitments to clients about what can be delivered when without any idea about what might actually need to be done to make it happen. Schedule slippages are virtually guranteed under Waterfall because there is little opportunity to “guide” a project back on track without significant investments by either the contractor, client or both.
I've seen it happen scores of times over my career; and what's really remarkable is that the Dev Leads and other Sages keep on truckin', doing the same thing with the next project resulting in the usual catastrophic finale of tensions, dreaded “Change Requests” and contract clause enforcements. You know, Happy-Happy Joy-Joy stuff.
Third: Whiskey Tango Foxtrot? Nothing more to add here - it's plain delusional.
Fourth: I don't know even where to begin here so I'll relate an anecdotal story about a team I worked with who was into Year 2 of a Waterfall monstrosity that was behind schedule, laden with bugs and causing everyone involved an enormous amount of stress and lost holidays and weekends.
I was involved in the initial “Requirements Capture” exercises - long, drawn-out episodes sometimes lasting for 8 hours at a go where we tried to perfect what the system would/should look like. We had input from representatives of committees of users who said “this is how we want it to work”.
When I recently went to work on some bugs with the team I asked about these requirements docs:
“Oh those?”, came the reply from one developer.
“We abandoned almost all of them months ago because they were so out of date and didn't reflect what they wanted after the features went through QA”.
Game. Set. Match.
There are NO pros to the Waterfall method unless you are lucky enough to write software that requires no imagination beyond “insert tab A into slot B”. And that, in my humble opinion, is a fantasy of the first water.
If ever there was a patent example of why Big Design Up Front / Waterfall projects fail, it's this one. And, unlike your typical isolated corporate app, the failings of this one can result in more than just budget overruns: It could contribute to lives lost.
Today's Washington Post details the story of a multi-million dollar effort to revamp the FBI's criminal investigation tracking system, which was to be called the Virtual Case File (VCF). A contractor, Science Applications International Corp. (SAIC) was engaged to develop the system, which, after years of development and testing had collapsed upon itself in a heap of bugs that render the system unusable.
Reading this multi-page account is like watching a trainwreck unfold or the Titanic sinking: You just know it's going to fail because it's so obvious. It reeks of impending failure. Some hallmarks:
Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed.
"A month before delivery, you don't have SPRs," Azmi said. "You're making things pretty. . . . You're changing colors."
Ahoy, captain! Iceberg 12 o'clock high! Icceeeeberrrrrg!
The article also notes (laments?) the failures as stemming from “failures of almost every kind, including poor conception... muddled execution”. Sounds like a raging waterfall of doom to me. From here on, this story takes a bizarre set of twists and turns, with the FBI acquiescing to SAIC's demands to rewrite their system from scratch as opposed to modifying off-the-shelf products (hmm-- is that even feasible?)
But the evidence for this BDUF/watefall failure continued to mount, including launching the new software all at once with minimal testing, half-baked (sloughed) features and, of course, the infamous “sliding delivery date”:
By 2004, even as the news grew worse behind the scenes, FBI officials struggled to put an optimistic spin on their software upgrade.
In March, testifying before a House subcommittee, Mueller said that the FBI had experienced "a delay with the contractor" but that the problem had been "righted." He said he expected that "the last piece of Virtual Case File would be in by this summer."
Two months later, Azmi -- who had been named the bureau's chief information officer -- pushed back the estimate further, predicting that SAIC would deliver the product in December.
And of course, this resulted in “feuding over change orders, system requirements and other issues“. Game. Set. Match. Thanks for playing “I Want To Deliver Software with a Waterfall“. Please collect your kick in the rump as a parting gift.
Given the importance of an agency like the FBI, this multi-million dollar fiasco is really disturbing. Even more disturbing is that despite mountains of evidence to the contrary, major software firms continue to indulge the fallacy of the waterfall to deliver high-risk, critical software projects.
Epilogue
In the end, Lockheed Martin stepped in to develop a replacement for VCF called Sentinel, projected to cost $425M. Apparently Lockheed will be required to meet benchmarks (hmm, like Sprint Reviews?) and so far “has survived three review sessions and is on-budget and on-schedule”.
The takeaway from this cautionary tale is not to get into the same predicament: By carefully planning a segmenting work into smaller deliverable chunks that are continuously tested and reviewed by the client, fiascos like the VCF can begin to dissipate, costs controlled and software that provides real value delivered.
|