Intellectual Phases
A framework to think about software projects
Note: I’ve decided to change the format of this blog, writing more “original” content with no or few citations (though still mainly relying on what I read and watch, rather than from my own experience).
“Intellectual phases” is a framework to think about projects.
Activities on the project gradually change one another: conception, requirements, product design, UI design, features, architecture, detailed code and data structure design, coding (the order may vary a bit). They could be boiled down into three phases: discovery, invention, and implementation.
As the project progresses, the focus (work activity) shifts between the phases:
On a specific project, phases have a different level of inherent risk
On specific projects, systems, and products, different intellectual phases might be more challenging (risky, uncertain) than other phases. (We could also say that some phases have more essential complexity than others.)
For a project with the particularly challenging discovery, the requirements (and, maybe, the very definition of the product is not clear). Example: the project of leveraging data to improve Li-ion batteries. Or, think about Google+ product and why it failed.
On a project with challenging invention, it’s hard to get the design right. The “design” here may refer to system design, to UI, or to software design. An example of a project with a challenging system and software design: software deployed on a fleet of IoT devices that should manage the state of the devices and send the telemetry data to the cloud in the unreliable network. Or, think about project and task management software such as Jira and their struggles with effective system and UI design.
On a project with challenging implementation, it’s a problem to get the details right and achieve the required level of quality. Examples of such projects: a database engine, or life-critical software such as software for a pacemaker (which may not be a challenging problem itself, but ensuring that the code has zero bugs is).
A project with risky implementation and more certain discovery and invention could be visualised like this:
More than one phase might be challenging on a project. For example, a project with uncertain requirements and difficult-to-get-well design, but less challenging implementation:
Strategies to manage the different level of uncertainty of the intellectual phases on the project
Match work activities to the riskiness of the intellectual phases
Trade complexity between the intellectual phases
Allocate work so that more senior team members work in more risky areas and less senior ones work in less risky areas.
Match work activities to the riskiness of the intellectual phases
Matching work activities is the strategy of dealing with the varying level of the essential complexity of different intellectual phases on a specific project.
On projects with uncertain or challenging discovery, invention, or implementation, it makes sense to do more activities from the following lists:
Ideas for reinforcing risky discovery
Actually gather and manage user stories and requirements
Review requirements
Do user studies, involve users
Prototype
Try to verify ideas before implementation
Do shorter product iterations
Brainstorm
(Can you think of more ways? Please share!)
Ideas for reinforcing risky invention
Actually do deliberate design work (perhaps, in pairs or groups)
Review design and architecture (RFCs)
Use an RFC template, design/architecture review checklists
Prototype UI
Try to verify the design before implementation, build Proof-of-Concept
Do shorter design/architecture iterations, avoid Big Design Up Front
Document the decisions that don’t warrant a full RFC
(Can you think of more ways? Please share!)
Ideas for reinforcing risky implementation
Program in pairs, review code
Use a PR template, code review checklists
Automate tests, make testing and debugging feedback loop shorter
Use Test-Driven Development (TDD)
Use static analysis
Write regression tests, benchmarks
Use a strict coding standard
(Can you think of more ways? Please share!)
The common threads between the ideas for reinforcing discovery, invention, and implementation are:
Review the results: requirements, design, code
More formal process and/or structure: formal reviews, templates, checklists, code standard
Verify before implementation: prototype the product, build a proof-of-concept, use TDD
Do shorter iterations in an uncertain space: Scrum, evolutionary architecture
To do more work in more challenging phases, it’s necessary to do less work in less challenging phases.
Trade complexity between the intellectual phases
Some of the Intellectual phases on the project might have more essential complexity (uncertainty, risk) than the project has resources to take. This might be, for example, having not enough people to work in the challenging area or a client not tolerating huge variability in the delivery date due to the risks on the project.
It could be possible to trade some uncertainty between the phases to spread the risks more evenly across them.
Trade complexity between the user and the product
Manage the scope of the product. Decide what one type of users we want to please. Select the use cases and the problems that we want to solve for users, and what we don’t. Let the users find by themselves creative ways to satisfy advanced needs which the product has decided not to try to solve. Example: the Unix philosophy.
Trade complexity between discovery and invention
Trade complexity between the intellectual phases of discovery and implementation by managing the features and the non-functional requirements for the system. For example, deciding that the system should have many unrelated features will puzzle the designers about combining them into a coherent whole.
For system’s non-functional requirements, consider the CAP theorem: it’s easier to design a distributed system that is just consistent, available, or partition-tolerant than a system that is consistent and available (CA), or consistent and partition-tolerant (CP), and it’s simply impossible to “invent” a system that combines all three these qualities, no matter how much we want to “discover” it.
Can you think of more interesting examples? Please share them!
Trade complexity between invention and implementation
There are several ways of how one can Trade complexity between the intellectual phases of invention (design, architecture) and implementation.
One way is to achieve some quality of the system either by making all constituent parts to have it or by clever design of the system that makes the desired quality an emergent property of the system. For example, a database can achieve durability by demanding all its components (tables, indexes) to be durable (thus the implementation complexity is imposed on them), or by having a clever design with a write-ahead-log that allows other components to be non-durable, which is easier to implement.
The UI (frontend) can employ some design techniques to improve the user’s perception and mask the deficiencies of the underlying implementation. For example, if the backend system is slow to load the results, the frontend can cache the last viewed version and present it to the user with a mark that these results may be outdated.
Can you think of more ways to trade the complexity between design and implementation? Please share them!
Related ideas
Focus on reviewing the risky part of the task
Review the riskiest parts of the code
Focus on the riskiest parts of the system
Focus on the riskiest qualities of the system
Do the riskiest thing first
Prototype to buy information
Verify ideas before implementation
Do shorter iterations in an uncertain space
Use checklists
Delay architecture decisions as much as possible
Log design decisions on two levels
Reduce the product’s scope (Implement only top 20% features)
Focus on a handful of qualities for a product or system
References
Earl Beede. 10x software development
Steve McConnell. Understanding software projects
Sidney Dekker. Drift into Failure
Andrew Crotty et al. The case for interactive data exploration accelerators (IDEAs)