A Time-Based Regression Matched Markets Approach for Designing Geo Experiments
Abstract
Although randomized controlled trials are regarded as the "gold standard" for causal inference, advertisers have been hesitant to embrace them as their primary method of experimental design and analysis due to technical difficulties in implementing them in the online advertising context. To help mitigate some of these challenges while still providing the rigor of a randomized controlled trial, Vaver and Koehler (2011) introduced the concept of a "geo experiment." However, it may not always be possible to rely on randomization when designing a geo experiment. For example, it may not be realistic to expect randomization to create balanced experimental groups when some of the geos are markedly different from all of the others or when there are only a few geos available for experimentation. In addition, randomization may not always be feasible given some of the specific requirements that advertisers often must impose on their experiments in practice---such as the need to run a smaller scale geo experiment within a given budget or the need to include certain geos in specific experimental groups. Consequently, advertisers may sometimes prefer to forgo some of the benefits of randomization, and in this paper we introduce a more systematic "matched markets" approach that, subject to the advertiser's constraints, greedily searches for experimental group assignments that appear to satisfy some of the critical assumptions of the "Time-Based Regression" (TBR) model for analyzing geo experiments that was introduced in Kerman et al. (2017). If the modeling assumptions of TBR do indeed hold, then the experimental designs that are recommended by our matched markets approach lead to straightforward causal estimates.
Research Areas
