Exploring the Limits of Out-of-Distribution Detection

Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-training can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% (current SOTA) to 77%. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained models are well-suited to outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. We observe similar trends on genomics, achieving 85% with just 1 example per OOD class. For multi-modal image-text pre-trained models such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information (without any accompanying images) and show that this outperforms previous SOTA on several standard OOD benchmark tasks.

• Machine Intelligence