“Overlapping Experiment Infrastructure at Google” D. Tang
Published KDD Proceedings 2010
This paper describes the thought process and concepts behind the extensive testing philosophy and infrastructure at Google.
Reading log: This is a very useful paper I read a while ago and dug up again for a client in June. The concepts I learned here seem to emerge intermittently when meeting with clients.
I think this should be required reading for anyone getting started with overlapping testing infrastructures (those that manage multiple tests at the same time). Lean Analytics!
Key take-aways include:
- the concept of domains, subsets and layers to partition parameters and design infrastructure
- binary push vs data push; separating testing parameters from program code.
- Canary experiments and defining expected range of monitored metrics
My concerns (i.e. interests or applications in mind) with re-reading this paper for my client were:
- Applying overlapping infrastructure to A/B testing vs. Multi-Arm Bandit testing,
- The particulars of having a shared control group
- Using such an infrastructure to test and select machine learning algorithm hyperparameters