DataComp for Vision-Language Models

Matteo Farina
Vishaal Udandarao
Thao Nguyen
Selim Kuzucu
Maximilian Böther
Andreas Hochlehnert
Adhiraj Ghosh
Marianna Nezhurina
Karsten Roth
Joschka Struber
Sebastian Dziadzio
Yuhui Zhang
Elaine Sui
Soumya Jahagirdhar
Dhruba Ghosh
Hasan Hammoud
Thomas De Min
Simone Caldarella
Jehanzeb Mirza
Sedrick Keh
Mehdi Cherti
Hilde Kuehne
Bernt Schiele
Serena Yeung
Ferjad Naeem
Ana Klimovic
Elisa Ricci
Matthias Bethge
Sewoong Oh
Ameya Prabhu
Jenia Jitsev
Massimiliano Mancini
Ludwig Schmidt
Nikhil Parthasarathy
(2026)

Abstract

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at this https URL.

Research Areas

×