Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation

We consider the task of effect estimation of resource allocation algorithms through clinical trials. Such algorithms are tasked with optimally
utilizing severely limited intervention resources,
with the goal of maximizing their overall benefits
derived. Evaluation of such algorithms through
clinical trials proves difficult, notwithstanding the
scale of the trial, because the agents’ outcomes are
inextricably linked through the budget constraint
controlling the intervention decisions. Towards
building more powerful estimators with improved
statistical significance estimates, we propose a
novel concept involving retrospective reshuffling
of participants across experimental arms at the
end of a clinical trial. We identify conditions
under which such reassignments are permissible
and can be leveraged to construct counterfactual
clinical trials, whose outcomes can be accurately
‘observed’ without uncertainty, for free. We prove
theoretically that such an estimator is more accurate than common estimators based on sample
means — we show that it returns an unbiased estimate and simultaneously reduces variance. We
demonstrate the value of our approach through
empirical experiments on both, real case studies
as well as synthetic and realistic data sets and
show improved estimation accuracy across the
board.

• Human-Computer Interaction and Visualization

• Machine Intelligence