If you're reading this, you're most likely interested in knowing what agile is about, what it can bring to the table for your team, what it can mean to your clients and ultimately why it is so damned better than the way you're doing things now. More than likely, you may have come to the conclusion that the way you're doing things now may need some improvement.
Whatever your reasons, welcome to the first installment of a multi-part series where I'll be exploring the concepts of agile software development and two "poster kids" for agile methodologies, Scrum and XP. My intent with these articles is to provide a no-nonsense, cut-to-the chase perspective on the ins and outs of agile and how to begin charting a course to them from your current development methods. Along the way, we're going to examine some best practices and tools for implementing them; we'll also look at some scenarios for building out an agile team in your organization and how to get the ball rolling.
We have a lot of ground to cover, so let's get started!
What is Agile?
As the old song goes, we need to 'begin the beguine'. Agile didn't fall out of the sky (despite what some may think), and it wasn't the brainchild of marketing executives or jaded software snakeoil salesmen. It came about as a decided reaction to the increasing failure of software projects to meet expectations in cost, schedule, scope and quality. As IT research specialists, The Standish Group, note in a 1994 report software projects at the time were exercises in failure:
...Research shows a staggering 31.1% of projects will be canceled before they ever get completed. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg...
On the success side, the average is only 16.2% for software projects that are completed on-time and on-budget. In the larger companies, the news is even worse: only 9% of their projects come in on-time and on-budget. And, even when these projects are completed, many are no more than a mere shadow of their original specification requirements. Projects completed by the largest American companies have only approximately 42% of the originally-proposed features and functions. Smaller companies do much better. A total of 78.4% of their software projects will get deployed with at least 74.2% of their original features and functions.
- The CHAOS Report (1994), The Standish Group
Sound familiar? It should, because the story hasn't changed all that much today. Much of the reason behind the failures then as now can be traced to the prevailing practices that govern software development, such as the infamous waterfall model, which enforces top-heavy bureaucratic processes that cascade through a set series of phases from requirements analysis to design to implementation to testing to integration and finally maintenance.
While adequate from a strict engineering perspective, the rigidity of the waterfall precludes it from responding to change. In fact, change is anathema to waterfall projects. The ideal situation is for requirements to be crystallized at the beginning of development and never changed again. Because each phase is dependent on input from the previous one, it is inherently brittle and any slippages have a way of becoming compounded and magnified as they wind their way down.
This inflexibility ultimately leads to lengthy debugging sessions because no time or means is provided to test software until the end. Unsurprisingly, delays and cost overruns ensue as the team becomes embroiled in epic struggles to deliver on their commitments (most of which were made without them). Clearly, something needed to change and for the better.
In February 2001 change was in the wind as seventeen software professionals who would come to be known as the Agile Alliance, met at a ski resort in Utah to hammer out what would become the seminal document for the agile software development movement, The Agile Manifesto (http://agilemanifesto.org/).
Their goal was to help uncover better ways of developing software and to take these measures and use them to help others. What they produced, like the concept of agile itself, is simple and concise, embodying four key values, and twelve principles.
Now, to be sure, 2001 does not mark the year that agile was born but it does represent its turning point in history. Many of its practices are in fact much older and represent best-of-breed. Some of its tenets were inspired by just-in-time manufacturing processes like lean and Sashimi.
Nonetheless, the Agile Manifesto represents a unique common view by a group of diverse founding "agilists", including: Kent Beck (XP), Ward Cunningham (design patterns, wikis), Andrew Hunt (Pragmatic Programmer), Dave Thomas (Pragmatic Programmer), Jeff Sutherland (Scrum), Mike Beedle (Scrum), Ken Schwaber (Scrum), Robert Martin (ObjectMentor, patterns and practices), and Ron Jeffries (XP).
From their experiences and knowledge these industry heavyweights distilled agile software development practices into four key values:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
At first glance, these values do not seem very radical at all, but at the time it was near-heresy. In fact, even now if you mention these ideas to grizzled project managers and bosses, they will probably go on a tear. Working software over a 300-page binder? Inconceivable! Customer collaboration instead of lengthy, protracted contract negotiations? Blasphemy! Responding to change instead of following a project plan!?!? Don't let the door hit you on the way out.
How can such seemingly contradictory and non-sensical values help software teams at all? Principally, by enabling teams to focus on customer satisfaction through their actions as opposed to their processes. While it may seem that delivering tonnes of documentation and use cases and UML diagrams is valuable, when it begins to take up more time than delivering tested and working software it is pointless. Adhering to an inflexible and unworkable plan, while admirable in the British sense of "over the top, boys!", only serves to punish a team that cannot possibly deliver. The same goes for dogmatic faith in overblown software tools and processes.
Thus, the emphasis for agile is on practices that deliver concrete business value to customers over practices that do not.
The Agile Principles
From these four core agile values the Agile Alliance enumerated twelve agile principles[2] to exemplify what agile practices would/should be (from their perspective) and are every bit as controversial as the values themselves. Again, to most this just seems practical common-sense, but they are almost counter-intuitive for most managers. I won't enumerate them all here, but instead focus on a several examples:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- This is a definining hallmark of agile software development. With testing and integration occurring daily, working software is delivered in small increments, allowing clients to watch their product come to life.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
- Martin Fowler talks about this very point in his essay, The New Methodology. Because testing and integration occurs frequently, making changes shouldn't be seen as a negative. A client's priorities can shift, so provide the processes and means to allow clients to change their minds, stay competitive and still get great, working software.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Again, another hallmark of agile software. This is all about shrinking the usual delivery scale by several orders of magnitude. By focusing on delivering only as much functionality as can fit in a shorter time frame, teams are empowered to release working software frequently and regularly.
- Working software is the primary measure of progress.
- What I consider the FUNDAMENTAL principle of all agile processes. If you're not delivering a working solution regularly, progress measurements are as indeterminate as fixing jell-o to the wall.
- Simplicity--the art of maximizing the amount of work not done--is essential.
- This principle embodies what I call the Zen of Agile, and can be best seen in the practices that Dave Thomas and Andrew Hunt catalogued in their landmark text, The Pragmatic Programmer. For example, they define concepts like the DRY principle (Don't Repeat Yourself), YAGNI (You Ain't Gonna Need It), writing "Good Enough" code and leveraging our PCs to automate repetitive processes. The emphasis here is on doing only as much as you need to and no more. Gold-plating software leads to delays, unnecessary complexity and confusion for the team. Manually cranking through processes that could be automated takes unnecessary time and effort and can cause more errors than it may solve. Similarly, avoid re-inventing components or classes for common implementations -- look for opportunities to leverage components or code that has been written earlier to promote reuse.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- Another controversial measure: Teams that are allowed to self-organize can and do in fact produce better systems. In contrast to prevailing wisdom, agile methodologies such as Scrum and XP do not compartmentalize developers in “vertical“ definitions that work in isolation of each other. Much like a military platoon or special ops unit, they are cross-functional. Specialists tackle jobs in their domain, but everyone on the team contributes toward meeting the objective, even if a task may fall outside a member's domain. Agile team members help each other meet their goals.
Agile in a Nutshell
It's said that some of the greatest books, treatises and essays in Western European history were conceived of at a time of crisis, often inspired by or in reaction to the events of the day. For J.J. Rousseau, it was the French Revolution and the French Republic; for Friedrich Hayek, creeping socialism and the rise of Keynesian economics; for H.D. Thoreau, slavery and civil disobedience.
Similarly, the Agile Alliance and thus the Agile Manifesto were created in reaction to the crises in the software industry that resulted from the following of inflexible and dogmatic methodologies such as the waterfall model which are better suited to building bridges than software. It squarely contests the conventional wisdom of software project management that tries to force time-consuming and counter-productive processes on development teams in favour of one that frees teams to do what they do best and in a manner that enables them some control over their part in the project.
Is it the silver bullet that will cure all that ails the software world? Not quite; as much as it represents freedom for teams, it also represents a discipline, and like all disciplines it takes some time to get right. And it carries some risk. As one of the inventors of Scrum, Ken Schwaber, notes in a recent podcast, of all the companies that attempt to implement his agile methodology, only one third will ever make it.[3] However, with risk also comes reward.
Irrespective of the agile methodology you may choose for your next project, the fundamentals remain the same: To be agile is to be customer, not process-driven; to value adaptation over rigid prediction; to value people over processes. It may be relatively new, but it has a pedigree that comes from years of experience. It is the distillation of the key ingredients for successfully delivering software with limited time and resources.
What's Next?
In my next installment, we'll look at a leading agile methodology that puts agile practices and values in play and has even garnered interest in the hallowed halls of Microsoft's Redmond Campus: Scrum.
[1] The Agile Alliance (http://www.agilealliance.org/)
[2] Principles behind the Agile Manifesto (http://agilemanifesto.org/principles.html)
[3] Agile Software Development with Scrum Podcast Series - Scrum FAQ Part 3 (http://blogs.conchango.com/howardvanrooijen/archive/2005/11/05/2362.aspx)