Google Research

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications through Visual Programming

  • Ruofei Du
  • Na Li
  • Jing Jin
  • Michelle Carney
  • Scott Joseph Miles
  • Maria Kleiner
  • Xiuxiu Yuan
  • Yinda Zhang
  • Anuva Kulkarni
  • Xingyu "Bruce" Liu
  • Ahmed K Sabie
  • Sergio Orts Escolano
  • Abhishek Kar
  • Ping Yu
  • Ram Iyengar
  • Adarsh Kowdle
  • Alex Olwal
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), ACM (to appear)

Abstract

In recent years, there has been a proliferation of multimedia applications that leverage machine learning (ML) for interactive experiences. Prototyping ML-based applications is, however, still challenging, given complex workflows that are not ideal for design and experimentation. To better understand these challenges, we conducted a formative study with seven ML practitioners to gather insights about common ML evaluation workflows.

This study helped us derive six design goals, which informed Rapsai, a visual programming platform for rapid and iterative development of end-to-end ML-based multimedia applications. Rapsai is based on a node-graph editor to facilitate interactive characterization and visualization of ML model performance. Rapsai streamlines end-to-end prototyping with interactive data augmentation and model comparison capabilities in its no-coding environment. Our evaluation of Rapsai in four real-world case studies (N=15) suggests that practitioners can accelerate their workflow, make more informed decisions, analyze strengths and weaknesses, and holistically evaluate model behavior with real-world input.

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work