Toronto

Imagen is a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. It uses large Transformer language models for text embeddings and the strength of diffusion models in high-fidelity image generation.

Pix2Seq is a simple, generic framework for object detection, based on the intuition that if a neural net knows about where and what the objects are, we just need to teach it how to read them out. Pix2Seq v2 solves a set of vision tasks in a single unified sequence interface.

Self-supervised learning has shown to be useful in pre-training for two medical image tasks: dermatology condition classification and multi-label chest X-ray classification. Large self-supervised models are robust to distribution shift and learn well with little labeled data.

Denoising diffusion models have shown to be effective for high-fidelity image generation. We introduce image super-resolution by repeated refinement steps (SR3) and cascaded diffusion models (CDM) for state-of-the-art class conditional ImageNet generation.

Voyager is a novel neural network for data prefetching. Voyager represents a significant step toward practical neural prefetching with online training and full coverage of the address space, while reducing computational overhead by 15-20× and storage overhead by 110-200×.

Models fine-tuned from the same initialization lie in the same loss basin. Model soups, constructed by averaging the weights of multiple models fine-tuned with different hyperparameter configurations, provide better accuracy and robustness than any individual model.

LOLNeRF is a method for learning a generative 3D model based on neural radiance fields, trained solely from data with only single views of each object. With LOLNeRF one can generate new 3D views of an image without requiring multi-view data for training or at test time.

SimCLR is a simple framework for self-supervised representation learning on natural images. It advances the state-of-the-art in self-supervised and semi-supervised learning, achieving a new record in image classification on ImageNet with limited labeled data.

Palette is a simple and general framework for image-to-image translation, formulated in terms of conditional denoising diffusion models. We demonstrate state-of-the-art results on a suite of four tasks: image inpainting, uncropping, colorization, and JPEG artifact removal.

We propose a self-supervised capsule architecture that canonicalizes data while simultaneously decomposing point clouds into parts to perform unsupervised representation learning.

Our team develops and applies representational similarity techniques to understand how different design choices affect how neural networks process their inputs. Our most recent work explores similarities and differences between Vision Transformer and ResNet models.