• 510.205.3525

  • hello at getvariant.com

Experiment Lifecycle

Any experimentation system has to provide two disparate capabilities: instrumentation and analysis. While discussions of the statistical analysis dominate the social media and the literature, the instrumentation is commonly if unwittingly thought of as an afterthought easily attainable by a rank-and-file developer. Nothing could be farther from the truth.

The Three Tenets of Experimentation System Design

In their influential 2017 paper Aleksander Fabijan et al postulate the three paramount requirements for any experimentation service. I consider these in detail and see how Variant stacks up and even challenges some of the assumptions that seem inherent in these principles.