Sparse Additive Generative Models of Text
Abstract
Generative models of text typically associate
a multinomial with every class label or topic.
Even in simple models this requires the estimation of thousands of parameters; in multi-faceted latent variable models, standard approaches require additional latent \switching" variables for every token, complicating
inference. In this paper, we propose an alternative generative model for text. The central idea is that each class label or latent
topic is endowed with a model of the deviation in log-frequency from a constant background distribution. This approach has two
key advantages: we can enforce sparsity to
prevent over tting, and we can combine generative facets through simple addition in log
space, avoiding the need for latent switching
variables. We demonstrate the applicability
of this idea to a range of scenarios: classification, topic modeling, and more complex
multifaceted generative models.
a multinomial with every class label or topic.
Even in simple models this requires the estimation of thousands of parameters; in multi-faceted latent variable models, standard approaches require additional latent \switching" variables for every token, complicating
inference. In this paper, we propose an alternative generative model for text. The central idea is that each class label or latent
topic is endowed with a model of the deviation in log-frequency from a constant background distribution. This approach has two
key advantages: we can enforce sparsity to
prevent over tting, and we can combine generative facets through simple addition in log
space, avoiding the need for latent switching
variables. We demonstrate the applicability
of this idea to a range of scenarios: classification, topic modeling, and more complex
multifaceted generative models.
