Why Agile practices are especially helpful in complex domains
Short iterations, frequent reflection, pair programming, autonomous teams, frequent integration
In this post, I use the results from the complex systems theory to show how some Agile development practices are particularly helpful when you work on a complex system or a project in a complex domain.
Some of the connections lay on the surface (such as that short iterations help to explore the a priori unknown complex space quicker), yet some other connections are less apparent.
By “complex system” I assume the following definition, stripped-down version from Drift into Failure:
a) The system is open, affecting and affected by the environment;
b) The system consists of many different components and agents, interacting in many different ways;
c) Interactions in the system can be non-linear. Small events can produce large results.
c) It’s hard or impossible to predict how the system will respond to certain changes in the structure or actions of the agents.
By “complex domain” or “complex space” I mean a large, sprawling socio-technical complex system, and a project in this domain that would benefit from Agile practices (as per the post below) could be some part of this system. The example could be the software part (the project) for a self-driving car, within a larger complex domain of autonomous car driving. Or, on a smaller scale, even writing some platform tool or library in a company with many dev teams. The company, the developers and the code they write would be the “complex space” in this case.
The system (project) complexity is not a binary quality but rather a gradient. Not all of the software projects and system are very complex, but I would say that most of the modern software projects are complex enough for the ideas below in this post to be applicable to them.
Short iterations
Shorter iterations mean quicker probing and quicker feedback.
It’s generally unknowable how the complex system will respond to some action (e. g. a new feature), therefore, complex systems always remain unknown in a certain sense. Shorter iterations accelerate the rate of early discovery and thus minimize the risk that major rework is needed down the line.
Frequent reflection
Complex systems are prone to drift into failure. One way to reduce the risk of stepping beyond the safety boundary is making smaller steps (iterations) and learn continuously.
Complex systems always change, therefore, we need to refresh our assumptions and learnings about the system regularly, to scrutinize earlier conclusions.
Work in pairs
Each engineer views the complex system uniquely because everyone is an agent in the system, nobody is outside it. It’s impossible to determine whose view is right and whose view is wrong because the agents effectively live in different environments. Therefore, when two or more engineers design a feature or work on a problem in a complex system, they always can arrive at a better solution that each of them could on their own.
Diversity of opinion helps to prevent exhaustive optimisation of the system (when the lack of slack and margins can turn small perturbations into large events).
Autonomous teams
The flip side of that everyone working on a complex project look at it from a different angle is that consensus is often impossible. Arguing about the “shared” parts of the system causes a lot of tension between people.
Frequent integration
Small changes can fail the complex software in unpredictable ways, so we need to integrate and test the system as a whole frequently to reduce the defect removal gap.
The Cynefin framework perspective
Another way to look at the relationship between Agile and complexity is the Cynefin framework.
The action loop for the Complex domain in Cynefin is Probe—Sense—Respond. This action loop matches the Scrum process:
Probe: During the sprint, implement something self-sufficient, reasonably complete (in case of systems with a UI—a working, or at least a visible feature).
Sense: Gather feedback from users or engineers in the adjacent areas. Retrospect the previous sprint.
Respond: Plan the next sprint. Implement new features.
Beware of “quick and dirty” solutions: complex systems have memory
To be objective, I have to note one Agile practice which can actually be harmful in a complex space: getting some not well thought-through solution out of the gate as quickly as possible to gather feedback can have negative side effects that are hard to remove afterwards. For example, one library function initially developed poorly can make other developers adapt their code or write workarounds. Then, the original piece cannot be fixed easily because then these workarounds would break. This exact situation happens frequently in SDKs of programming languages and popular libraries such as Java, PHP, jQuery, C++, etc. The “complex space” in this example is the developer community as a whole and all the code written using the given technology.
References
Sidney Dekker. Drift Info Failure
Complicated systems consist of a lot of parts and interactions but are closed, unlike complex which are open.
Dave Snowden. Complex Adaptive Systems:
When you architect a complex system, you design it for early failure. Because only through early failure could you discover what’s possible anyway.
Most human systems are complex.
If you are in a complex world, you have to manage by creating boundary conditions, running experiments, and responding to what happens.
You can’t understand a complex system through modelling or analysis. You can only understand it by interaction with real-time feedback loops over multiple agents so that you don’t have cognitive bias.
You want to do smaller experiments in parallel much faster so that bad unintended consequences you can destroy, and the good ones you can amplify. [Reminds me of Chaos Engineering]
Architect for discovery before you architect for delivery. If you start with delivery and skip on the discovery, you’re going to miss opportunities as well as threats.
Steve McConnell. Understanding Software Projects