Intellectual Phases

A framework to think about software projects

Note: I’ve decided to change the format of this blog, writing more “original” content with no or few citations (though still mainly relying on what I read and watch, rather than from my own experience).


“Intellectual phases” is a framework to think about projects.

Activities on the project gradually change one another: conception, requirements, product design, UI design, features, architecture, detailed code and data structure design, coding (the order may vary a bit). They could be boiled down into three phases: discovery, invention, and implementation.

As the project progresses, the focus (work activity) shifts between the phases:

On a specific project, phases have a different level of inherent risk

On specific projects, systems, and products, different intellectual phases might be more challenging (risky, uncertain) than other phases. (We could also say that some phases have more essential complexity than others.)

For a project with the particularly challenging discovery, the requirements (and, maybe, the very definition of the product is not clear). Example: the project of leveraging data to improve Li-ion batteries. Or, think about Google+ product and why it failed.

On a project with challenging invention, it’s hard to get the design right. The “design” here may refer to system design, to UI, or to software design. An example of a project with a challenging system and software design: software deployed on a fleet of IoT devices that should manage the state of the devices and send the telemetry data to the cloud in the unreliable network. Or, think about project and task management software such as Jira and their struggles with effective system and UI design.

On a project with challenging implementation, it’s a problem to get the details right and achieve the required level of quality. Examples of such projects: a database engine, or life-critical software such as software for a pacemaker (which may not be a challenging problem itself, but ensuring that the code has zero bugs is).

A project with risky implementation and more certain discovery and invention could be visualised like this:

More than one phase might be challenging on a project. For example, a project with uncertain requirements and difficult-to-get-well design, but less challenging implementation:

Strategies to manage the different level of uncertainty of the intellectual phases on the project

  • Match work activities to the riskiness of the intellectual phases

  • Trade complexity between the intellectual phases

  • Allocate work so that more senior team members work in more risky areas and less senior ones work in less risky areas.

Match work activities to the riskiness of the intellectual phases

Matching work activities is the strategy of dealing with the varying level of the essential complexity of different intellectual phases on a specific project.

On projects with uncertain or challenging discovery, invention, or implementation, it makes sense to do more activities from the following lists:

Ideas for reinforcing risky discovery

  • Actually gather and manage user stories and requirements

  • Review requirements

  • Do user studies, involve users

  • Prototype

  • Try to verify ideas before implementation

  • Do shorter product iterations

  • Brainstorm

  • (Can you think of more ways? Please share!)

Ideas for reinforcing risky invention

  • Actually do deliberate design work (perhaps, in pairs or groups)

  • Review design and architecture (RFCs)

  • Use an RFC template, design/architecture review checklists

  • Prototype UI

  • Try to verify the design before implementation, build Proof-of-Concept

  • Do shorter design/architecture iterations, avoid Big Design Up Front

  • Document the decisions that don’t warrant a full RFC

  • (Can you think of more ways? Please share!)

Ideas for reinforcing risky implementation

  • Program in pairs, review code

  • Use a PR template, code review checklists

  • Automate tests, make testing and debugging feedback loop shorter

  • Use Test-Driven Development (TDD)

  • Use static analysis

  • Write regression tests, benchmarks

  • Use a strict coding standard

  • (Can you think of more ways? Please share!)


The common threads between the ideas for reinforcing discovery, invention, and implementation are:

  • Review the results: requirements, design, code

  • More formal process and/or structure: formal reviews, templates, checklists, code standard

  • Verify before implementation: prototype the product, build a proof-of-concept, use TDD

  • Do shorter iterations in an uncertain space: Scrum, evolutionary architecture

To do more work in more challenging phases, it’s necessary to do less work in less challenging phases.

Trade complexity between the intellectual phases

Some of the Intellectual phases on the project might have more essential complexity (uncertainty, risk) than the project has resources to take. This might be, for example, having not enough people to work in the challenging area or a client not tolerating huge variability in the delivery date due to the risks on the project.

It could be possible to trade some uncertainty between the phases to spread the risks more evenly across them.

Trade complexity between the user and the product

Manage the scope of the product. Decide what one type of users we want to please. Select the use cases and the problems that we want to solve for users, and what we don’t. Let the users find by themselves creative ways to satisfy advanced needs which the product has decided not to try to solve. Example: the Unix philosophy.

Trade complexity between discovery and invention

Trade complexity between the intellectual phases of discovery and implementation by managing the features and the non-functional requirements for the system. For example, deciding that the system should have many unrelated features will puzzle the designers about combining them into a coherent whole.

For system’s non-functional requirements, consider the CAP theorem: it’s easier to design a distributed system that is just consistent, available, or partition-tolerant than a system that is consistent and available (CA), or consistent and partition-tolerant (CP), and it’s simply impossible to “invent” a system that combines all three these qualities, no matter how much we want to “discover” it.

Can you think of more interesting examples? Please share them!

Trade complexity between invention and implementation

There are several ways of how one can Trade complexity between the intellectual phases of invention (design, architecture) and implementation.

One way is to achieve some quality of the system either by making all constituent parts to have it or by clever design of the system that makes the desired quality an emergent property of the system. For example, a database can achieve durability by demanding all its components (tables, indexes) to be durable (thus the implementation complexity is imposed on them), or by having a clever design with a write-ahead-log that allows other components to be non-durable, which is easier to implement.

The UI (frontend) can employ some design techniques to improve the user’s perception and mask the deficiencies of the underlying implementation. For example, if the backend system is slow to load the results, the frontend can cache the last viewed version and present it to the user with a mark that these results may be outdated.

Can you think of more ways to trade the complexity between design and implementation? Please share them!

Related ideas

  • Focus on reviewing the risky part of the task

  • Review the riskiest parts of the code

  • Focus on the riskiest parts of the system

  • Focus on the riskiest qualities of the system

  • Do the riskiest thing first

  • Prototype to buy information

  • Verify ideas before implementation

  • Do shorter iterations in an uncertain space

  • Work in pairs

  • Use checklists

  • Delay architecture decisions as much as possible

  • Log design decisions on two levels

  • Reduce the product’s scope (Implement only top 20% features)

  • Focus on a handful of qualities for a product or system

  • Emergence

References

Grady Booch on Twitter

Earl Beede. 10x software development

Steve McConnell. Understanding software projects

Sidney Dekker. Drift into Failure

Andrew Crotty et al. The case for interactive data exploration accelerators (IDEAs)