Engineering Ideas #5
Choosing Cloud DBMS, organisational bottlenecks, radical candor, software architecture, design docs
Choosing A Cloud DBMS: Architectures and Tradeoffs
Junjay Tan et al. validated that serverless Big Data computing platforms like Amazon Athena (Azure Data Lake Analytics, Google BigQuery) not only provide operational advantages, but also more scalable, cheaper, and often faster than more traditional cluster processing engines like Amazon Redshift, Presto, etc.
Authors provide simple rules of thumb for choosing architecture elements for systems in modern clouds:
As analytic (OLAP) applications move to the cloud, […] we identify following patterns and design choices that are advantageous: prioritizing low-cost object stores like S3 for data storage, using system-agnostic yet still performant columnar formats like ORC that allow easy switching to other systems for different workloads and making features that benefit subsequent runs like query precompilation and caching remote data to faster storage optional rather than required because they disadvantage ad hoc queries.
Work is Work
Coda Hale suggests concrete tactics to prevent the organization’s performance from stagnating:
As an organization hires more employees, work on productivity improvements must be a constant priority. Internal tooling, training, and services must be developed and fielded to ensure that all members are able to work on problems of continuously increasing impact. The ceaseless pursuit of force multipliers is the only possible route to superlinear productivity improvements as an organization grows.
An interesting hint:
If an organization is largely working on the same types of problems it was in previous years, it’s cause for concern. Teams dedicated to internal tooling should be staffed and given the explicit direction of building tools and optimizing processes to help increase their co-workers’ productivity.
Code Hale also talks about organizational bottlenecks (e. g. in the form of people or groups communication with whom is on the critical workflow path of many other people in the company), which reminds me of several Edmond Lau’s ideas from The Effective Engineer:
Take a holistic view of your iteration loop. Don’t ignore any organizational and team-related bottlenecks that may be within your circle of influence.
Build shared ownership of code. Increase your bus factor to be greater than one so that you’re not a bottleneck for development. This will give you the flexibility to focus on other high-leverage activities.
Radical Candor
By Rina Artstain
Radical Candor is about building relationships where the truth is welcome in order to get better quality results and improve personally.
The last part of the post about design reviews reminded me of many ideas about meetings and decision making from Peter Drucker’s The Effective Executive.
This can be a debate only or a debate & decide meeting, but whichever you choose, you need to make sure to allocate a separate time for deciding and be clear about when you are transitioning into decide mode. There’s nothing less efficient than trying to make a decision when some people are still debating.
The Myth of Architect as Chess Master
By Ben Northrop
It's important to think about software architecture as a process and not a destination. […] To truly know the right next "move" requires first understanding the rules of the game you're in. What are the business drivers, risks, technical constraints, legacy components, integrations, possible options, trade-offs, and so on.
The expectation for the architect should be to setup the framework or process by which good architectural "moves" are made (Michael Keeling describes this well in Design It).
Scaling Engineering Teams via Writing Things Down and Sharing - aka RFCs
Gergely Orosz shares an easy to apply yet a very helpful process for sharing the technical knowledge across the company and improving the quality of the technical designs:
First, the lack of visibility on others building or having built the same thing as my team. Second, the tech and architecture debt accumulated due to different teams building things very differently, both approach-wise and quality-wise.
There is a way to tackle both these issues pretty well:
Do planning before building something new.
Capture this plan in a short, written document.
Have a few, select people approve this plan before starting work. Similar to how it's a good quality gate to only merge a pull request after someone does a review on it, it makes a huge difference if before starting work on a project, a few, relevant people validate the planned work. These can be senior engineers, people on a team who will use the feature and so on.
Send this planning document out to all engineers in the company and let anyone and everyone comment on it.
Have everyone follow the above steps for every project that is beyond some super trivial complexity and iterate on a lightweight, RFC-like process so it works well for your org or company.
As unlikely it might sound, the above process both works and scales really well, from a handful of engineers to teams of thousands.
Pretty much everything could be improved by writing:
Writing and sharing that writing with others creates accountability. It also almost always leads to more thorough decisions. A simple way to increase code quality? Do code review in writing, before merging. A simple way to have a meeting be less of a waste of time? Have a written agenda before the meeting, then write up and send out decisions and actions afterwards. A simple way to run projects with fewer surprises? Have the team write down what they are planning to do and share it with others.
This makes me think about Stephen Covey’s “Begin with the end in mind” from “The 7 Habits of Highly Effective People”.
You can subscribe to new Engineering Ideas via RSS (e. g. using Feedly) or e-mail.