Analogy between model regularisation and the engineering design process
The regularisation algorithm described in "Variational Auto-Regularized Alignment for Sim-to-Real Control":
Simulate inputs
Train VAE's decoder on the simulated inputs
Train VAE's encoder with the pre-trained decoder on real inputs
Take the updated simulation parameters from the encoder and repeat from
step 1.
This algorithm is analogous to the engineering design process emphasising simplicity:
Collect (or try to predict, i. e. "simulate") the requirements.
Design the system (or the API) to serve these requirements well, without thinking about the implementation details.
Implement the system and put it into production.
Adjust the requirements based on the feedback and the real usage patterns and the experience of deploying the system in production, extract new requirements, and repeat from step 1.
In "Bumper-Sticker API Design", Joshua Bloch pointed to these steps, too:
When designing an API, first gather requirements—with a healthy degree of skepticism.
Structure requirements as use-cases.
Keep APIs free of implementations details.
Consider the performance consequences of API design decisions, but don't warp an API to achieve performance gains.
Simple design is more reliable than complex design for the same reason as regularised models are more reliable than overfitted models.
This analogy came to me upon thinking about the Principle of Computational Equivalence.