As promised to the attendees (all 10 or so of them…!) to my riveting and speculative session at the Agile 2008 conference yesterday, below is the link to my slide deck in PowerPoint 97–2003 format:

Agile 2008 – Who’s Teaching Best Practices?

I’ll write more on the feedback I received – it was definitely an engaging conversation that was lively and went a half-hour over the “official” closeout time.  The majority (all?) of attendees were either professors or masters/phD students who worked with professors at institutions from across Canada, the US and Europe.  Basically things broke down into two broad categories:

a) Methodology – I hung a lantern on this one:  Yes, looking at course catalogs or calendars is highly speculative and not “scientific”.  What was more interesting, however, were comments that suggested that descriptions are deliberately and intentionally vague in many institutions so as to afford instructors and professors flexibility to teach topics not specifically outlined.  It was also mentioned that the process for changing course descriptions was so difficult (read: bureaucratic) that it was best to go with the most bland and inoffensive wording so as to avoid dealing with the gatekeepers.

b) Role of the University as Developer of Leaders not Technique Instructors – This one caught me completely off-guard as I didn’t expect it from folks who are idenitifying as “pro-agile”.  There were a lot of comments in this area which essentially distill to “it’s not our job to prepare graduates for working with a specific technology or technique – we teach them how to think and lead”.  The corollary to this is that this is what colleges and “practical” technical institutes should do.

Although I understand where this is coming from, I’ve got a few issues with this response, as I am sure many readers might:  There were no apologies made for the elitist tone.  While I agree there is a need for CS/SE students to be able to learn how to think in abstractions (Joel Spolsky writes on this in his blog), there is also a compatible reality that students should be able to apply the abstract to some concrete tasks – very much what agile/iterative/lean processes do and their corresponding best practices.

I found it completely foreign to hear that there was absolutely no interest in teaching courses specific to best practices.  They could be taught as a component of a course, but it was felt that to add this on to the typical coursework would be unworkable.

Time to think things through…

I acknowledged that my methods were imperfect, my conclusions very rough – however, my personal experiences with university grads as interns suggested to me that there was a potentially interesting connection between what students were learning and what was actually expected of them in the real world.

Attendees posited to me that if industry wants this to change, they need to pony up more cash and reinstate or bolster their own internal training courses.  In effect, the responsibility for teaching agile/iterative/lean SDLC and best practices like refactoring code so that it’s readable and of value is not in the domain of the university.  Never should be, never will be.

Despite this reaction, I’ve been given a lot of good feedback, and I have to take time to process it and begin to address some points more thoroughly, perhaps in a future posting.

Food for thought from Google:

Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day. This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections.

Whoa.

Earlier today, while reading a posting on an internal Microsoft DL dedicated to agile shop talk, I found a link to an Engineering Excellence wiki dedicated to Scrum.  Whoa!  It appears that my favourite agile/lean/iterative process has roots going back at least to 2004 within the company and that folks were seriously discussing its merits as a delivery model.

One section of the wiki dealing with the Theory of Constraints as applied within Scrum immediately caught my attention:  Within it was a link to a page detailing The Five Orders of Ignorance.  This sounded very Zen to me.  It also conjured up images of kung-fu warriors trying to attain enlightenment from their master as they ascended the steps of their craft.

But that’s just me.

In reality, the Five Orders describe how much we do or do not know about a problem, with the understanding that different processes are more or less appropriate fro addressing problems with differing degrees of uncertainty.  At first glance, this seems quite in simpatico with Boehm/McConnell’s observations with the Cone of Uncertainty around problem domains and when it is most appropriate to estimate a task.

The Five Orders of Ignorance were written eight years ago by Philip G. Armour in an article he wrote for The Communications of the ACM (Association for Computing Machinery) by the same name.  In it, he defines them as:

The 0th Order: Lack of Ignorance

• When you know something and can demonstrate your lack of ignorance through some tangible form.

The 1^st Order: Lack of Knowledge

• When you do not know something and can readily identify the fact.

The 2^nd Order: Lack of Awareness

• When you do not know that you do not know something.  In other words, you’re not only ignorant of something, but you are also unaware of the fact.

The 3^rd Order: Lack of Process

• When you do not know a suitably efficient way to discover that you are unaware of something.

The 4^th Order: Meta Ignorance

• When you do not know about the Five Orders of Ignorance.  This is very Douglas Adams, as having just read this bullet, you now know that there are Five Orders of Ignorance.  Don’t panic.

So now I know what I should know, but didn’t know and in some cases can’t know since I didn’t know I didn’t know in the first place.  Now what?

Using the Five Orders, we can begin to classify the unknowns and uncertainties and how we manage them within the context of a project.  A few years back, the press and late night personalities had a hay-day with a comment that Donald Rumsfeld made with respect to the activities of insurgents in Iraq and Afghanistan:

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.

After hearing the preceding sentences, the last one sounded positively hilarious:  How on Earth could there be unknown unknowns?  Bwa-hahahaha.  However, in reality this is just an expression of the 2^nd Order of Ignorance.  Note that this doesn’t mean “ignorant” in the pejorative sense – this is about the recognition that more information is needed and that we’re operating within the context of low-knowledge.

Gee.  Sounds an awful lot like most software/IT projects, doesn’t it?

The anti-pattern to 2OI is when we choose to do nothing about it, which takes us to the 3^rd Order of Ignorance – this is probably best expressed in BDUF/waterfall projects where we make an explicit statement about not knowing the unknowns and really have no plan for addressing them because we actively deny the existence of unknown unknowables through rigid project phases.  The project plan is the plan.  And it is all-knowing.  The plan is the Holiest of artifacts and must never be questioned, etc., etc., etc.

But it gets worse: If an unknown unknowable should legitimately come our way and intrude on our carefully-crafted MS Project Plan, we treat it like a bugblatter beast of Traal and promptly wrap a towel around our head in the belief that if we can’t see it, it can’t see us.

And so the madness goes ever on – the condition of low-knowledge persists, paradoxically in the face of the very things that could dispel the “ignorance” and take us closer down the ladder to the 0th Order. 

The Five Orders of Ignorance provide us with an indication of how we address wicked projects and their inherent problems:  Knowing that we can’t know everything about a problem domain and that we need an adequate process to manage this “ignorance” is the key to successful delivery.  This is a fundamental underpinning for all agile/lean/iterative processes:  Ignorance isn’t viewed as a contractual failing (ie. “what do you mean schedule/cost is slipping because you didn’t realize ‘x’ – you should have known ‘x’ !) but rather as an expected opportunity for gathering new knowledge about the domain.

So, we all know that MSFT acquired Norway-based FAST Search & Transfer (OSE:FAST) on April 25 this year, and that the aquisition would add their powerful Enterprise Search Platform (ESP) to its stable of high-end search offerings for customers.  Since then, there’s been a lingering question of how long it would take to complete the integration of ESP into MOSS.

Wonder no more, there’s a collection of web parts to facilitate an early take on this integration over on CodePlex, as part of the Search Community Toolkit:  FAST ESP Web Parts for SharePoint Server 2007

The package provides the following web parts:

• Search Box -- Web part that supports creation of a search box for query term submission
• Result List -- Web part that displays search results and supports sorting, pagination, and navigator-based filtering
• Navigators -- Web part for displaying dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks 
• Breadcrumb -- Web part for displaying the search term(s) and list of navigators used to obtain the current result set 
• Search Site Template – Template for linking together the various Web parts to build a complete search site in one step

As with all things on CodePlex, there is a certain amount of caveat emptor, however the source code is available and they do encourage improvements and suggestions for feature enhancements.  As Martha would say, this is a good thing.  FAST knows their stuff – if you’re working on any enterprise-scale MOSS deployments, eventually you’ll be working with FAST.

I’ve been preoccupied with helping my client assess and visualize a SharePoint 2003 to MOSS migration for the past few weeks, and as a result my blogging suffers.  Always something!

Today, a couple of agile resources I wanted to post for posterity and share.  First, a white paper by Kent Beck published by Microsoft on the use of tools in agile software development entitled Tools for Agility:

This is an excellent, quick essay by the creator of eXtreme Programming on the value of tooling for enabling agile software development.  A consistent theme for Beck is how the evolution of utilities like continuous integration and xUnit frameworks foster faster transitions between tasks that an agile/iterative/lean team encounters over the lifespan of the project.  Beck also offer his prognostications on future agile tools and practices.

Next up, the Alt.NET Podcast.  Yes, I’m a Johnny-come-lately to discovering this feed, but I’m a huge fan of developer podcasts like Hanselminutes and the Agile Toolkit as a means of obtaining insightful information during my down-time driving to the cottage or enduring a ride on the TTC subway (uggh!).

This morning, I listened to the Adopting Agile episode with Owen Rogers – really good stuff that made me think hard about the philosophies and positions I’ve held about non-agile teams and how to persuade them to make the transition gradually with just a single commitment to deliver a working increment of software once a month.  From here, in Rogers’ experience, it’s easier to begin rationlizing the adoption of agile/lean/iterative practices to enable the once-a-month rhythm.  Very much in the “stone soup” tradition.

That’s it for today – I’m off to the cottage this evening for the “holiday long weekend” here in Canada.  Time to catch up on some Inversion of Control podcasts…