Google Research

Performance tournaments with crowdsourced judges

  • Daryl Pregibon
  • Williiam D Heavlin
Proceedings of the American Statistical Association, section on marketing statistics, American Statistical Association, 732 North Washtington Street, Alexandria, VA 22314-1943 (2013)


A performance slam is a competition among a fixed set of performances whereby pairs of performances are judged by audience participants. When performances are recorded on electronic media, performance slams become amenable to audiences that watch on­line and judge asynchronously (“crowdsourced”). In order to better entertain the audience, we want to show the better performances (“exploitation”). In order to identify the good videos, we want to glean a least some information about all videos (“exploration”).

Our approach has three elements: (1) We take our preference model from Bradley and Terry (1952). (2) Its parameters we calculate by rewriting the likelihood gradient into a fixed point estimate, one which mimics the estimate of Mantel and Haenszel (1959). (3) Each pair of performances is chosen sequentially, always chosen to minimize the weighted variance of (the logarithms of) the Bradley-Terry parameter estimates. Our preferred weights consist of the log­rank weights proposed by Savage (1956).

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work