Google Research

Predicting Human Performance in Vertical Menu Selection Using Deep Learning

  • Yang Li
  • Samy Bengio
  • Gilles Bailly
CHI'18 (2018) (to appear)


Predicting human performance in interaction tasks allows designers or developers to understand the expected performance of a target interface without actually testing it with real users. In this work, we present a deep neural net to model and predict human performance in performing a sequence of UI tasks. In particular, we focus on a dominant class of tasks, i.e., target selection from a vertical list or menu, which resembles many interaction scenarios in modern user interfaces. We experimented with our deep neural net using a public dataset collected from a desktop laboratory environment and a dataset collected from hundreds of touchscreen smartphone users via crowdsourcing. Our model significantly outperformed previous methods in various settings. Importantly, our method, as a deep model, can easily incorporate additional UI attributes such as visual appearance and content semantics without changing model architectures—these attributes were hard to capture with previous methods. We discussed our insights into the behaviors of our model.

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work