Scalable Inference in Latent Variable Models

Latent variable techniques are pivotal in tasks ranging from
predicting user click patterns and targeting ads to organiz-
ing the news and managing user generated content. La-
tent variable techniques like topic modeling, clustering, and
subspace estimation provide substantial insight into the la-
tent structure of complex data with little or no external
guidance making them ideal for reasoning about large-scale,
rapidly evolving datasets. Unfortunately, due to the data
dependencies and global state introduced by latent variables
and the iterative nature of latent variable inference, latent-
variable techniques are often prohibitively expensive to ap-
ply to large-scale, streaming datasets.
In this paper we present a scalable parallel framework
for ecient inference in latent variable models over stream-
ing web-scale data. Our framework addresses three key
challenges: 1) synchronizing the global state which includes
global latent variables (e.g., cluster centers and dictionaries);
2) eciently storing and retrieving the large local state which
includes the data-points and their corresponding latent vari-
ables (e.g., cluster membership); and 3) sequentially incor-
porating streaming data (e.g., the news). We address these
challenges by introducing: 1) a novel delta-based aggrega-
tion system with a bandwidth-ecient communication pro-
tocol; 2) schedule-aware out-of-core storage; and 3) approxi-
mate forward sampling to rapidly incorporate new data. We
demonstrate state-of-the-art performance of our framework
by easily tackling datasets two orders of magnitude larger
than those addressed by the current state-of-the-art. Fur-
thermore, we provide an optimized and easily customizable
open-source implementation of the framework.

• Machine intelligence