Autoregressive styled text image generation, but make it reliable

The task of generating images of styled text (often referred to as Styled Handwritten Text Generation - HTG) is attracting attention from different communities, including Computer Vision, Document Analysis, and Graphic Design. A lot of research effort in this task is dedicated to developing strategies that reproduce the stylistic characteristics of a given writer, with impressive results in terms of adherence and generalization achieved by the recently proposed Autoregressive Transformer paradigm for HGT. However, faithfulness to the desired text to render is not yet guaranteed, often due to collapse, generation artifacts, and lack of controllability in autoreressive models.
In this work, we extend the autoregressive formulation by framing HTG as a multimodal prompt-conditioned generation task and tackle the content controllability issues by introducing text input tokens for better alignment with the visual ones and by devising a Classifier-Free-Guidance-based training strategy for our autoregressive model. Through extensive experimental validation, we demonstrate that our approach, dubbed Eruku, not only can generalize well to unseen styles but can also better follow the textual prompt to achieve content faithfulness.