A Comparison of Causal Inference Methods for Estimating Sales Lift

In this paper, we compare a variety of methods for causal inference through simulation, examining their sensitivity to and asymptotic behavior in the presence of correlation between (heterogeneous) treatment effect size and propensity to be treated, as well as their robustness to model mis-specification. We limit our focus to well-established methods relevant to the estimation of sales lift, which initially motivated this paper and serves as an illustrative example throughout. We demonstrate that popular matching methods often fail to adequately debias lift estimates, and that even doubly robust estimators, when naively implemented, fail to deliver statistically valid confidence intervals. The culprit is inadequate standard error estimators, which often yield insufficient confidence interval coverage because they fail to take into account uncertainty at early stages of the causal model. As an alternative, we discuss a more reliable approach: the use of a doubly robust point estimator with a sandwich standard error estimator.

• Algorithms and Theory