Experiment Design Kit

Overview#

Most failed experiments are decided before the first observation: too few participants, the wrong randomisation unit, or repeated peeking at the results. This toolkit covers those design decisions for both A/B tests and controlled trials.

What it does#

• Planning: effect sizes, sample size and power for proportions, two-sample and Welch t-tests, and minimum detectable effect.
• Bayesian planning: power and sample size for threshold or ROPE decisions, by simulation.
• Variance reduction: CUPED for continuous metrics.
• Sequential inference: mSPRT always-valid p-values and alpha-spending bounds for optional stopping.
• Randomisation: stratified, permuted-block, cluster (with ICC-aware power) and switchback designs, with balance diagnostics.
• Checks: sample ratio mismatch tests and multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini-Hochberg and others).

Usage#

experiment-design-kit sample-size --method proportion --p1 0.10 --p2 0.12
experiment-design-kit power --method two-sample --d 0.5 --n 64
experiment-design-kit mde --method two-sample --n 64 --sd 10

Engineering#

Dependency-light, with tests and a seedable outcome simulator for checking each design before running it.