Diane Tang
Authored Publications
Sort By
Focus on the Long-Term: It's better for Users and Business
Henning Hohnhold
Deirdre O'Brien
Proceedings 21st Conference on Knowledge Discovery and Data Mining, ACM, Sydney, Australia (2015)
Preview abstract
Over the past 10+ years, online companies large and small have adopted widespread A/B testing as a robust data-based method for evaluating potential product improvements. In online experimentation, it is straightforward to measure the short-term effect, i.e., the impact observed during the experiment. However, the short-term effect is not always predictive of the long-term effect, i.e., the final impact once the product has fully launched and users have changed their behavior in response. Thus, the challenge is how to determine the long-term user impact while still being able to make decisions in a timely manner.
We tackle that challenge in this paper by first developing experiment methodology for quantifying long-term user learning. We then apply this methodology to ads shown on Google search, more specifically, to determine and quantify the drivers of ads blindness and sightedness, the phenomenon of users changing their inherent propensity to click on or interact with ads.
We use these results to create a model that uses metrics measurable in the short-term to predict the long-term. We learn that user satisfaction is paramount: ads blindness and sightedness are driven by the quality of previously viewed or clicked ads, as measured by both ad relevance and landing page quality. Focusing on user satisfaction both ensures happier users but also makes business sense, as our results illustrate. We describe two major applications of our findings: a conceptual change to our search ads auction that further increased the importance of ads quality, and a 50%
reduction of the ad load on Google’s mobile search interface.
The results presented in this paper are generalizable in two major ways. First, the methodology may be used to quantify user learning effects and to evaluate online experiments in contexts other than ads. Second, the ads blindness/sightedness results indicate that a focus on user satisfaction could help to reduce the ad load on the internet at large with long-term neutral, or even positive, business impact.
View details
Overlapping Experiment Infrastructure: More, Better, Faster Experimentation
Ashish Agarwal
Deirdre O'Brien
Mike Meyer
Proceedings 16th Conference on Knowledge Discovery and Data Mining, ACM, Washington, DC (2010), pp. 17-26
Preview abstract
At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments, how to run experiments that produce better decisions, and how to run them faster. In this paper, we describe Google’s overlapping experiment infrastructure that is a key component to solving these problems. In addition, because an experiment infrastructure alone is insufficient, we also discuss the associated tools and educational processes required to use it effectively. We conclude by describing trends that show the success of this overall experimental environment. While the paper specifically describes the experiment system and experimental processes we have in place at Google, we believe they can be generalized and applied by any entity interested in using experimentation to improve search engines and other web applications.
The presentation is available online.
View details
Session Viewer: Visual Exploratory Analysis of Web Session Logs
Daniel M. Russell
Tamara Munzner
Symposium on Visual Analytics Science and Technology (VAST), IEEE (2007), pp. 147-154
Preview abstract
Large-scale session log analysis typically includes statistical methods and detailed log examinations. While both methods have merits, statistical methods can miss previously unknown subpopulations in the data and detailed analyses may have selection biases. We therefore built Session Viewer, a visualization tool to facilitate and bridge between statistical and detailed analyses. Taking
a multiple-coordinated view approach, Session Viewer shows
multiple session populations at the Aggregate, Multiple, and Detail data levels to support different analysis styles. To bridge between the statistical and the detailed analysis levels, Session Viewer provides fluid traversal between data levels and side-by-side comparison at all data levels. We describe an analysis of a large-scale web usage study to demonstrate the use of Session Viewer, where we quantified the importance of grouping sessions based on task type.
View details
Query logs alone are not enough
Preview
Daniel Russell
WWW 2007 Workshop on Query Log Analysis: Social and Technological Changes
