Scrum, Agile, Complexity and Ants

I started re-reading Agile Project Management with Scrum by Ken Schwaber a few days ago, after not having looked at it in far too long. I was struck by the similarities between Schwaber’s view on the complexities of software development and the concepts of Complex Systems Theory I recently read while helping someone do research for a paper. I have to think that Schwaber and the other founders of Scrum were at least partly inspired by it.

Complex Systems Theory is, not surprisingly, difficult to describe in a nutshell. A complex system is a system composed of actors that interact with and influence each other in many different ways. It is essentially a network with nearly infinite relationships between the actors. Natural ecosystems are good examples, where millions of animals, plants, bacteria and fungus interact and affect each other in uncountable ways.

Complexity Science studies how this variety of actors and interactions can result in “spontaneous self-organization.” The system organizes itself, adapts, and evolves without anyone directing or planning the organization. The adaptation is a result of actors constantly adapting to each other. The international stock markets dropping in reaction to a news report is an example. No one person held a meeting of all the world’s stock brokers and said “SELL SELL SELL!” Some read the news and adjusted their portfolios. This lead to others adjusting their portfolios and so on. Another example from nature is a community of ants changing to a new food source. The queen ant didn’t send out royal directives. One ant found a new food source and interacted with some other ants and then your picnic basket was overrun.

The study of complex systems has also been applied to business, in particular to management and the engineering and manufacturing of products, as a way to encourage innovative thinking and rapid responses to change. Traditional management relies on the idea that the business environment is predictable, and so business strategy and product development can be planned months ahead and instructions can be passed down the hierarchy. But anyone trying to manage a requirements document in a software development project will tell you a different story. The modern business world is unpredictable, fast moving, and incredibly complex. Complexity science tells us that if given the right environment, the people in an organization and their relationships can spontaneously organize and adapt to change much faster and more successfully than a hierarchically controlled system. Thus Agile works because it is empirical and adaptive, not a defined process. The team experiments, learns and validates continually rather than adhering to a restrictive and sometimes inappropriate process.

Re-reading Schwaber’s book reminded me that with the complexity of software development, success requires the self-organization of the team. I often get lost in the ‘process’ of scrum, the sprints and scrum meetings and backlogs and reviews, forgetting that scrum is almost an anti-process. The ‘process’ of scrum is simply creating as little process to dictate behavior as possible, and instead ensuring the optimum environment for the team to self-organize. Scrum is about removing impediments and distractions and enhancing communication and collaboration. The sprints and reviews are not intended to tell the ants how to get to the picnic basket. They’re there to enable the the team to find a faster and better way to the picnic basket, even if someone keeps moving it.

About Andy Hayward

Software Lifecycle Geek and blogger, student of Agile, avid reader and motorcycling nut.
This entry was posted in Agile Development, Complexity & Compliance Standards, Integrity, Scrum, Software Product Lines and tagged , , , , . Bookmark the permalink.

4 Responses to Scrum, Agile, Complexity and Ants

  1. PM Hut says:

    Hi Andy,

    There are many influential bloggers in the project management community who are claiming that Scrum is failing/fading. What do you think of that?

    They are also saying that it’s being replaced by Kanban (see from scrum to kanban).

    Do you really think that Scrum will still exist in a few years from now? And why do you think we have all this schisms when it comes to Agile?

    • Andy Hayward says:

      That’s a lot of questions to answer!

      Scrum was first described formally in 2001, but it is based on principles first described in the 1980s, so it’s been around in some form or another for 25 years. In that time it has evolved, grown and faded. Whether it is fading or not now I don’t know, but I do know many organizations who are looking at it and attempting to adopt it or some of its principles.

      We have schisms in the Agile community for the same reason we have had schisms in any community. People have different experiences, work in different environments with different teams and using and developing different technologies, and as a result people have different experiences with the different agile frameworks. Some of these people are very passionate about what they do, and that results in friction. Get three developers into a room to discuss the best way to solve any complex problem and you’ll have 3 ‘schisms’ then too.

      The biggest problem with the idea of Scrum disappearing is that it’s based on the assumption that Scrum is prescriptive and a silver bullet for problematic software projects and teams. Scrum and Kanban are frameworks for improving, not step-by-step guides to smooth development, and they are not mutually exclusive. In your example, the team felt more comfortable with Kanban, possibly because they weren’t able to apply all of the guidelines of Scrum, probably for good reasons. However, other business stakeholders may have been more comfortable with the idea of regular sprints than the standard WIP signboard of Kanban, and in future your team may have to adapt the process again. The important part is to regularly review what’s working and what’s not, and when something isn’t working, look for things to adopt, change, or remove.

      As such, considering the community’s healthy discussion, what’s more likely is that Scrum and Kanban (and Lean and XP etc) will evolve and become a collection of ‘tools’ that can be applied as needed.

  2. Pingback: Scrum, Agile, Complexity and Ants | Why are requirements so hard?

  3. Ken Schwaber says:

    You are right on target, and I was heavily influenced by complexity theory. Self-organization and empiricism are the two foundations of Scrum with which complexity is addressed.
    Keep up the good blogs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s