Why use Agile Project Management?

Agile is an iterative and incremental software development methodology where requirements and solutions evolve through collaborative processes.

The Waterfall Process is often blamed for failed Software Projects.

The process of developing a new computer system was often lengthy and expensive and they often suffered from delays, budget overruns and complete failure due to cancellations or because the finished system didn't deliver what was expected. In time, the cause of many of the failures was seen to be the process itself.

Waterfall consists of a series of stages that are completed sequentially with each stage producing outputs that became the input to the next. The initial System Analysis and Design used various prescriptive techniques to produce a specification for a team of programmers to implement in code which when complete would be tested, deployed and then maintained.

classic waterfall methodology stages - analysis, design, implementation, testing, deployment, maintenance

Can you spot the problem? Yes - water flows downhill! It wasn't easy for later stages to provide input or valuable feedback into previous ones. Although there was typically some overlap the specialists responsible for a given stage were often moved on to their next project once their stage was completed.

  • What if a requirement was missed and only identified during testing?
  • What if a deployment issue required changes to the system design?
  • What if testing highlighted a misunderstanding with requirements?

Cost of making changes increased dramatically

The cost of making a change increased dramatically as the consequences of any decision fed through to the other stages. If testing uncovered an incorrect requirement it meant revisiting the Analysis, Design and Implementation stages again - extra communication, effort and frequently negotiations.

If a system failed was it because the requirements were wrong? Or the design was faulty? Or maybe it was coded badly? Or not tested well enough? Or operations screwed up? Companies and teams resorted to detailed contracts and excessive documentation to avoid the risk of being 'blamed' for their contribution.

Because the process was lengthy and expensive and it wasn't easy to go back to a previous step there was a tendency to over-specify systems. Instead of asking for just what was required, people felt they needed to make the best of their one opportunity to request everything they could conceivably think of and this further compounded issues by making systems way more complex than was really necessary.

Even if everything was done perfectly, the systems often took so long to be completed that what was finally delivered was often the system that the company needed 2 years ago when the requirements were first identified but which no longer completely suited their needs.

Waterfall didn't allow companies to react

The waterfall approach simply didn't allow companies to react quickly enough to changing business conditions because the lead-time for delivering a system was so long and the outcome variable.

These system failures were so common that cartoons were created to satirise the issues which were well known for some time.

how software development projects really work and why they often fail

The basic premise of Agile is to maximize value - focus on the things that increase productivity and remove impediments.

Agile was conceived as a lightweight alternative to the established heavyweight Project Management methodologies that had been proven to be ineffective when applied to software development.

Based on the experiences of successful and productive teams and new software development practices such as Test Driven Development it was focused on removing unproductive activities and enabling skilled people to focus their efforts on what they did best.

Instead of approaching a computer system like an engineering construction (such as building a house) where the complete foundation is built first before the next layer is added, people recognized that software was different and much more pliable. Good software development techniques meant that code didn't need to be seen as static but should evolve as a system was developed. Now, all that needed to be built was just what was needed for the next feature.

Fixed SQL Database Schema vs NoSQL

The rigid and inflexible database schemas of Relational Database Management Systems (RDBMS) or 'SQL' were often viewed as the foundation for any new system.

Agile software development has contributed to the increased adoption of NoSQL database technologies such as MongoDB which promote flexibility and remove some of the need for lengthy up-front schema design work.

Agile was an enabler to allow teams to focus on delivering working features to the customer, often within the first weeks of a project and ongoing throughout it's lifecycle. This provided immediate feedback and allowed quick changes when necessary as the customer could both see visible progress and provide better direction for future work.

Waterfall Approach

traditional waterfall development fails to deliver anything usable of value until project completion

Despite the large expenditure of effort and cost, the system doesn't provide usable features until it is complete.

Agile Approach

agile development delivers usable features sooner and immediate business value

Features are usable as soon as they are completed delivering immediate value with much less overall effort.

The minimum amount of effort is required to create a "Minumum Viable Product (MVP) which finally allowed companies to try out ideas and react quickly. An iterative process is used to build the software incrementally with each increment being a working, usable version progressing toward the final goal.

Instead of work being handed from team to team of separate specialists an Agile team is cross-functional with all of the skills necessary to analyze, architect, design, build, test and deploy the solution.

However, while there are some recognized practices the Agile approach itself was not prescriptive and there were many flavors and variations. The core values and principles were enshrined in the Agile Manifesto.

The Agile Manifesto is based on a set of core values and principles.

The Agile Manifesto was the result of the leading proponents of various Agile approaches coming together and combining their ideas into a set of core values and principles for practitioners to follow.

Core Values

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Some claim agile does away with planning, contracts and documentation which it clearly doesn't. It simply places more value on things such as working software and collaboration as these are more likely to directly contribute to a successful outcome.

Twelve Principles of Agile

We follow these principles:

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  4. Business people and developers must work together daily throughout the project.
  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  7. Working software is the primary measure of progress.
  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  9. Continuous attention to technical excellence and good design enhances agility.
  10. Simplicity--the art of maximizing the amount of work not done--is essential.
  11. The best architectures, requirements, and designs emerge from self-organizing teams.
  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

The Agile Alliance was formed by many of the Agile Manifesto's authors to further promote and formalize the Agile approach.

Scrum gained the most traction and became synonymous with Agile.

Over time a variety of different approaches developed all based on the core values and principles of Agile. These included:

  • Agile Modeling
  • Agile Unified Process (AUP)
  • Crystal Clear
  • Crystal Methods
  • Dynamic Systems Development Method (DSDM)
  • Extreme Programming (XP)
  • Feature Driven Development (FDD)
  • Graphical System Design (GSD)
  • Kanban
  • Lean Software Development
  • Scrum

Scrum became the most widely known and recognized and gave birth to new titles within development teams such as 'Scrum Master'.

As recognition of Scrum as an approach to developing software grew there was an increasing need for more formal Scrum Master Certifications to identify those with the genuine skills and real experience who could introduce and champion it within a organization looking to adopt Agile.