Food for thought from Google:

Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day. This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections.

Whoa.

Earlier today, while reading a posting on an internal Microsoft DL dedicated to agile shop talk, I found a link to an Engineering Excellence wiki dedicated to Scrum.  Whoa!  It appears that my favourite agile/lean/iterative process has roots going back at least to 2004 within the company and that folks were seriously discussing its merits as a delivery model.

One section of the wiki dealing with the Theory of Constraints as applied within Scrum immediately caught my attention:  Within it was a link to a page detailing The Five Orders of Ignorance.  This sounded very Zen to me.  It also conjured up images of kung-fu warriors trying to attain enlightenment from their master as they ascended the steps of their craft.

But that’s just me.

In reality, the Five Orders describe how much we do or do not know about a problem, with the understanding that different processes are more or less appropriate fro addressing problems with differing degrees of uncertainty.  At first glance, this seems quite in simpatico with Boehm/McConnell’s observations with the Cone of Uncertainty around problem domains and when it is most appropriate to estimate a task.

The Five Orders of Ignorance were written eight years ago by Philip G. Armour in an article he wrote for The Communications of the ACM (Association for Computing Machinery) by the same name.  In it, he defines them as:

The 0th Order: Lack of Ignorance

• When you know something and can demonstrate your lack of ignorance through some tangible form.

The 1^st Order: Lack of Knowledge

• When you do not know something and can readily identify the fact.

The 2^nd Order: Lack of Awareness

• When you do not know that you do not know something.  In other words, you’re not only ignorant of something, but you are also unaware of the fact.

The 3^rd Order: Lack of Process

• When you do not know a suitably efficient way to discover that you are unaware of something.

The 4^th Order: Meta Ignorance

• When you do not know about the Five Orders of Ignorance.  This is very Douglas Adams, as having just read this bullet, you now know that there are Five Orders of Ignorance.  Don’t panic.

So now I know what I should know, but didn’t know and in some cases can’t know since I didn’t know I didn’t know in the first place.  Now what?

Using the Five Orders, we can begin to classify the unknowns and uncertainties and how we manage them within the context of a project.  A few years back, the press and late night personalities had a hay-day with a comment that Donald Rumsfeld made with respect to the activities of insurgents in Iraq and Afghanistan:

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.

After hearing the preceding sentences, the last one sounded positively hilarious:  How on Earth could there be unknown unknowns?  Bwa-hahahaha.  However, in reality this is just an expression of the 2^nd Order of Ignorance.  Note that this doesn’t mean “ignorant” in the pejorative sense – this is about the recognition that more information is needed and that we’re operating within the context of low-knowledge.

Gee.  Sounds an awful lot like most software/IT projects, doesn’t it?

The anti-pattern to 2OI is when we choose to do nothing about it, which takes us to the 3^rd Order of Ignorance – this is probably best expressed in BDUF/waterfall projects where we make an explicit statement about not knowing the unknowns and really have no plan for addressing them because we actively deny the existence of unknown unknowables through rigid project phases.  The project plan is the plan.  And it is all-knowing.  The plan is the Holiest of artifacts and must never be questioned, etc., etc., etc.

But it gets worse: If an unknown unknowable should legitimately come our way and intrude on our carefully-crafted MS Project Plan, we treat it like a bugblatter beast of Traal and promptly wrap a towel around our head in the belief that if we can’t see it, it can’t see us.

And so the madness goes ever on – the condition of low-knowledge persists, paradoxically in the face of the very things that could dispel the “ignorance” and take us closer down the ladder to the 0th Order. 

The Five Orders of Ignorance provide us with an indication of how we address wicked projects and their inherent problems:  Knowing that we can’t know everything about a problem domain and that we need an adequate process to manage this “ignorance” is the key to successful delivery.  This is a fundamental underpinning for all agile/lean/iterative processes:  Ignorance isn’t viewed as a contractual failing (ie. “what do you mean schedule/cost is slipping because you didn’t realize ‘x’ – you should have known ‘x’ !) but rather as an expected opportunity for gathering new knowledge about the domain.

So, we all know that MSFT acquired Norway-based FAST Search & Transfer (OSE:FAST) on April 25 this year, and that the aquisition would add their powerful Enterprise Search Platform (ESP) to its stable of high-end search offerings for customers.  Since then, there’s been a lingering question of how long it would take to complete the integration of ESP into MOSS.

Wonder no more, there’s a collection of web parts to facilitate an early take on this integration over on CodePlex, as part of the Search Community Toolkit:  FAST ESP Web Parts for SharePoint Server 2007

The package provides the following web parts:

• Search Box -- Web part that supports creation of a search box for query term submission
• Result List -- Web part that displays search results and supports sorting, pagination, and navigator-based filtering
• Navigators -- Web part for displaying dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks 
• Breadcrumb -- Web part for displaying the search term(s) and list of navigators used to obtain the current result set 
• Search Site Template – Template for linking together the various Web parts to build a complete search site in one step

As with all things on CodePlex, there is a certain amount of caveat emptor, however the source code is available and they do encourage improvements and suggestions for feature enhancements.  As Martha would say, this is a good thing.  FAST knows their stuff – if you’re working on any enterprise-scale MOSS deployments, eventually you’ll be working with FAST.