Jump to Content
Alex Bie

Alex Bie

https://alexbie98.github.io
Authored Publications
results

Publications

Years

Research Areas

Teams

Sort By
  • Title
  • Title, descending
  • Year
  • Year, descending
    On learnability of distribution classes with adaptive adversaries
    Tosca Lechner
    Alex Bie
    Gautam Kamath
    ICML 2025
    Preview abstract We consider the question of learnability of distribution classes in the presence of adaptive adversaries – that is, adversaries capable of inspecting the whole sample requested by a learner and applying their manipulations before passing it on to the learner. We formulate a general notion of learnability with respect to adaptive adversaries, taking into account the budget of the adversary. We show that learnability with respect to additive adaptive adversaries is a strictly stronger condition than learnability with respect to additive oblivious adversaries. View details
    Escaping Collapse: The Strength of Weak Data for Large Language Model Training
    Kareem Amin
    Sara Babakniya
    Alex Bie
    William Kong
    Sergei Vassilvitskii
    Umar Syed
    Advances in Neural Information Processing Systems 38 (NeurIPS 2025)
    Preview abstract Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples --- in much the same way that boosting focuses the efforts of the weak learner --- leads to improved performance. View details
    Private prediction for large-scale synthetic text generation
    Kareem Amin
    Alex Bie
    William Kong
    Alex Kurakin
    Natalia Ponomareva
    Umar Syed
    Andreas Terzis
    Sergei Vassilvitskii
    Findings of EMNLP 2024
    Preview abstract We present an approach for generating differentially private synthetic text using large language models (LLMs), via private prediction. In the private prediction framework, we only require the output synthetic data to satisfy differential privacy guarantees. This is in contrast to approaches that train a generative model on potentially sensitive user-supplied source data and seek to ensure the model itself is safe to release. We prompt a pretrained LLM with source data, but ensure that next-token predictions are made with differential privacy guarantees. Previous work in this paradigm reported generating a small number of examples (<10) at reasonable privacy levels, an amount of data that is useful only for downstream in-context learning or prompting. In contrast, we make changes that allow us to generate thousands of high-quality synthetic data points, greatly expanding the set of potential applications. Our improvements come from an improved privacy analysis and a better private selection mechanism, which makes use of the equivalence between the softmax layer for sampling tokens in LLMs and the exponential mechanism. Furthermore, we introduce a novel use of public predictions via the sparse vector technique, in which we do not pay privacy costs for tokens that are predictable without sensitive data; we find this to be particularly effective for structured data. View details
    Search on Google Scholar