Jasper Uijlings
I am a research scientist at Google working with Vittorio Ferrari on human-machine collaboration for large scale image annotation for both object detection and image segmentation.
I received my PhD in 2011 at the University of Amsterdam (UvA) under supervision of Prof. Dr.Ir. A. Smeulders and Prof. Dr. Ir. R. Scha. During this period, I was part of the UvA team which successfully participated in the PASCAL VOC Challenges, winning the classification challenge in 2008, receiving honorable mentions from 2009-2011, and winning object detection in 2012, the fi nal year of the competition. In 2011 we won the ILSVRC object detection challenge. From 2011-2013 I was a researcher at the University of Trento, Italy, where I worked on real-time video classification and on combining vision and language. From 2014-2015 I worked at the University of Edinburgh with Prof.Dr. Vittorio Ferrari on object boundary detection, weakly supervised object localisation, and efficient annotation for object detection.
Research Areas
Authored Publications
Google Publications
Other Publications
Sort By
Preview abstract
We address the problem of ensemble selection in transfer learning: Given a large pool of source models we want to select an ensemble of models which, after fine-tuning on the target training set, yields the best performance on the target test set. Since fine-tuning all possible ensembles is computationally prohibitive, we aim at predicting performance on the target dataset using a computationally efficient transferability metric. We propose several new transferability metrics designed for this task and evaluate them in a challenging and realistic transfer learning setup for semantic segmentation: we create a large and diverse pool of source models by considering 17 source datasets covering a wide variety of image domain, two different architectures, and two pre-training schemes. Given this pool, we then automatically select a subset to form an ensemble performing well on a given target dataset. We compare the ensemble selected by our method to two baselines which select a single source model, either (1) from the same pool as our method; or (2) from a pool containing large source models, each with similar capacity as an ensemble. Averaged over 17 target datasets, we outperform these baselines by 6.0% and 2.5% relative mean IoU, respectively.
View details
Transferability Estimation using Bhattacharyya Class Separability
Michal Pandy
Andrea Agostinelli
Vittorio Ferrari
CVPR (2022)
Preview abstract
Transfer learning has become a popular method for leveraging pre-trained models in computer vision. However, without performing computationally expensive fine-tuning, it is difficult to quantify which pre-trained source models are suitable for a specific target task, or, conversely, to which tasks a pre-trained source model can be easily adapted to. In this work, we propose Gaussian Bhattacharyya Coefficient (GBC), a novel method for quantifying transferability between a source model and a target dataset. In a first step we embed all target images in the feature space defined by the source model, and represent them with per-class Gaussians. Then, we estimate their pairwise class separability using the Bhattacharyya coefficient, yielding a simple and effective measure of how well the source model transfers to the target task. We evaluate GBC on image classification tasks in the context of dataset and architecture selection. Further, we also perform experiments on the more complex semantic segmentation transferability estimation task. We demonstrate that GBC outperforms state-of-the-art transferability metrics on most evaluation criteria in the semantic segmentation settings, matches the performance of top methods for dataset transferability in image classification, and performs best on architecture selection problems for image classification.
View details
How stable are Transferability Metrics evaluations?
Andrea Agostinelli
Michal Pandy
Vittorio Ferrari
ECCV (2022)
Preview abstract
Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without finetuning them all. However, existing works rely on custom experimental setups which differ across papers, leading to inconsistent conclusions about which transferability metrics work best. In this paper we conduct
a large-scale study by systematically constructing a broad range of 715k experimental setup variations. We discover that even small variations to an experimental setup lead to different conclusions about the superiority of a transferability metric over another. Then we propose better evaluations by aggregating across many experiments, enabling to reach more stable conclusions. As a result, we reveal the superiority of LogME at selecting good source datasets to transfer from in a semantic segmentation scenario, and N LEEP at selecting good source architectures in an image classification scenario. However, no single transferability metric works best in all scenarios.
View details
Preview abstract
Computer vision is driven by the many datasets available for training or evaluating novel methods. However, each dataset has a different set of class labels, visual definition of classes, images following
a specific distribution, annotation protocols, etc. In this paper we explore the automatic discovery of visual-semantic relations between labels across datasets. We aim to understand how instances of a certain class in a dataset relate to the instances of another class in another dataset. Are they in an identity, parent/child, overlap relation? Or is there no link between them at all? To find relations between labels across datasets, we propose methods based on language, on vision, and on their combination. We show that we can effectively discover label relations across datasets, as well as their type. We apply our method to four applications: understand label relations, identify missing aspects, increase label specificity, and predict transfer learning gains. We conclude that label relations cannot be established by looking at the names of classes alone, as they depend strongly on how each of the datasets was constructed.
View details
Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types
Michael Gygli
Vittorio Ferrari
Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2021)
Preview abstract
Transfer learning enables to re-use knowledge learned on a source task to help learning a target task. A simple form of transfer learning is common in current state-of-the-art computer vision models, i.e. pre-training a model for image classification on the ILSVRC dataset, and then fine-tune on any target task. However, previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood. In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains (consumer photos, autonomous driving, aerial imagery, underwater, indoor scenes, synthetic, close-ups) and task types (semantic segmentation, object detection, depth estimation, keypoint detection). Importantly, these are all complex, structured output tasks types relevant to modern computer vision applications.
In total we carry out over 2000 transfer learning experiments, including many where the source and target come from different image domains, task types, or both. We systematically analyze these experiments to understand the impact of image domain, task type, and dataset size on transfer learning performance. Our study leads to several insights and concrete recommendations: (1) for most tasks there exists a source which significantly outperforms ILSVRC'12 pre-training; (2) the image domain is the most important factor for achieving positive transfer; (3) the source dataset should \emph{include} the image domain of the target dataset to achieve best results; (4) at the same time, we observe only small negative effects when the image domain of the source task is much broader than that of the target; (5) transfer across task types can be beneficial, but its success is heavily dependent on both the source and target task types.
View details
Preview abstract
This paper proposes to make a first step towards compatible and hence reusable network components. Rather than training networks for different tasks independently, we adapt the training process to produce network components that are compatible across tasks. In particular, we split a network into two components, a features extractor and a target task head, and propose various approaches to accomplish compatibility between them. We systematically analyse these approaches on the task of image classification on standard datasets.
We demonstrate that we can produce components which are directly compatible without any fine-tuning or compromising accuracy on the original tasks. Afterwards, we demonstrate the use of compatible components on three applications: Unsupervised domain adaptation, transferring classifiers across feature extractors with different architectures, and increasing the computational efficiency of transfer learning.
View details
Training Neural Networks to Produce Compatible Features
Michael Gygli
Vittorio Ferrari
CVPR Workshop on Compositionality in Computer Vision (2020)
Preview abstract
This paper makes a first step towards compatible network components. We propose three ways which modify training to make components compatible: (i) We add a shared supervised auxiliary task which discriminates between the common classes. (ii) We add a shared self-supervised auxiliary task: rotation prediction. (iii) We initialize the networks using the same random weights.
On CIFAR-10 we show:
(i) we can train networks to produce compatible features, without degrading task accuracy compared to training the networks independently.
(ii) random initialization has a large effect on compatibility;
(ii) we can train incrementally: given previously trained components, we can train new ones which are also compatible with them
View details
Preview abstract
This paper aims to reduce the time to annotate images for panoptic
segmentation, which requires annotating segmentation masks and
class labels for all object instances and stuff regions. We formulate
our approach as a collaborative process between an annotator
and an automated assistant who take turns to jointly annotate an
image using a predefined pool of segments. Actions performed by
the annotator serve as a strong contextual signal. The assistant
intelligently reacts to this signal by annotating other parts of the
image on its own, which reduces the amount of work required by
the annotator. We perform thorough experiments on the COCO
panoptic dataset, both in simulation and with human annotators.
These demonstrate that our approach is significantly faster than
the recent machine-assisted interface of [4], and 2.4x to 5x faster
than manual polygon drawing. Finally, we show on ADE20k [62]
that our method can be used to efficiently annotate new datasets,
bootstrapping from a very small amount of annotated data.
View details
The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
Mohamad Hassan Mohamad Rom
Neil Alldrin
Ivan Krasin
Matteo Malloci
Vittorio Ferrari
IJCV (2020) (to appear)
Preview abstract
We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15x more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.
View details
Preview abstract
We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.
View details
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections
Theodora Kontogianni*
Michael Gygli*
Vittorio Ferrari
ECCV (2020) (to appear)
Preview abstract
In interactive object segmentation a user collaborates with a computer vision model to segment an object. Recent works employ convolutional neural networks for this task: Given an image and a set of corrections made by the user as input, they output a segmentation mask. These approaches achieve strong performance by training on large datasets but they keep the model parameters unchanged at test time. Instead, we recognize that user corrections can serve as sparse training examples and we propose a method that capitalizes on that idea to update the model parameters on-the-fly to the data at hand. Our approach enables the adaptation to a particular object and its background, to distributions shifts in a test set, to specific object classes, and even to large domain changes, where the imaging modality changes between training and testing. We perform extensive experiments on 8 diverse datasets and show: Compared to a model with frozen parameters, our method reduces the required corrections (i) by 9%-30% when distribution shifts are small between training and testing; (ii) by 12%-44% when specializing to a specific class; (iii) and by 60% and 77% when we completely change domain between training and testing.
View details
The Devil is in the Decoder: Classification, Regression and GANs
Zbigniew Wojna
Vittorio Ferrari
Nathan Silberman
Liang-chieh Chen
IJCV (2019) (to appear)
Preview abstract
Many machine vision applications require
predictions for every pixel of the input image (for exam-
ple semantic segmentation, boundary detection). Mod-
els for such problems usually consist of encoders which
decreases spatial resolution while learning a high-di-
mensional representation, followed by decoders who re-
cover the original input resolution and result in low-
dimensional predictions. While encoders have been stud-
ied rigorously, relatively few studies address the decoder
side. Therefore this paper presents an extensive com-
parison of a variety of decoders for a variety of pixel-
wise tasks ranging from classification, regression to syn-
thesis. Our contributions are: (1) Decoders matter: we
observe significant variance in results between different
types of decoders on various problems. (2) We introduce
new residual-like connections for decoders. (3) We in-
troduce a novel decoder: bilinear additive upsampling.
(4) We explore prediction artefacts.
View details
Preview abstract
We address interactive full image annotation, where the goal is to accurately segment all object and stuff regions in an image. We propose an interactive, scribble-based annotation framework which operates on the whole image to produce segmentations for all regions. This enables sharing scribble corrections across regions, and allows the annotator to focus on the largest errors made by the machine across the whole image. To realize this, we adapt Mask-RCNN into a fast interactive segmentation framework and introduce an instance-aware loss measured at the pixel-level in the full image canvas, which lets predictions for nearby regions properly compete for space. Finally, we
compare to interactive single object segmentation on the COCO panoptic dataset. We demonstrate that our interactive full image segmentation approach leads to a 5% IoU gain, reaching 90% IoU at a budget of four extreme clicks and four corrective scribbles per region.
View details
Preview abstract
We propose to revisit knowledge transfer for training
object detectors on target classes from weakly supervised
training images, helped by a set of source classes with
bounding-box annotations. We present a unified knowledge
transfer framework based on training a single neural net-
work multi-class object detector over all source classes, or-
ganized in a semantic hierarchy. This generates proposals
with scores at multiple levels in the hierarchy, which we use
to explore knowledge transfer over a broad range of gen-
erality, ranging from class-specific (bycicle to motorbike)
to class-generic (objectness to any class). Experiments
on the 200 object classes in the ILSVRC 2013 detection
dataset show that our technique (1) leads to much better
performance on the target classes (70.3% CorLoc, 36.9%
mAP) than a weakly supervised baseline which uses man-
ually engineered objectness [10] (50.5% CorLoc, 25.4%
mAP). (2) delivers target object detectors reaching 80% of
the mAP of their fully supervised counterparts. (3) outper-
forms the best reported transfer learning results [17, 42] on
this dataset (+41% CorLoc, +3% mAP). Moreover, we also
carry out several across-dataset knowledge transfer exper-
iments [25, 22, 32] and find that (4) our technique outper-
forms the weakly supervised baseline in all dataset pairs by
1.5 × −1.9×, establishing its general applicability.
View details
Preview abstract
We introduce Intelligent Annotation Dialogs for bound-
ing box annotation. We train an agent to automatically
choose a sequence of actions for a human annotator to pro-
duce a bounding box in a minimal amount of time. Specifi-
cally, we consider two actions: box verification [34], where
the annotator verifies a box generated by an object detector,
and manual box drawing. We explore two kinds of agents,
one based on predicting the probability that a box will be
positively verified, and the other based on reinforcement
learning. We demonstrate that (1) our agents are able to
learn efficient annotation strategies in several scenarios,
automatically adapting to the difficulty of an input image,
the desired quality of the boxes, the strenght of the detector,
and other factors; (2) in all scenarios the resulting annota-
tion dialogs speed up annotation compated to manual box
drawing alone and box verification alone, while also out-
performing any fixed combination of verification and draw-
ing in most scenarios; (3) in a realistic scenario where the
detector is iteratively re-trained, our agents evolve a series
of strategies that reflect the shifting trade-off between veri-
fication and drawing as the detector grows stronger.
View details
Preview abstract
Semantic classes can be either things (objects with a
well-defined shape, e.g. car, person) or stuff (amorphous
background regions, e.g. grass, sky). While lots of classifi-
cation and detection works focus on thing classes, less at-
tention has been given to stuff classes. Nonetheless, stuff
classes are important as they allow to explain important
aspects of an image, including (1) scene type; (2) which
thing classes are likely to be present and their location
(through contextual reasoning); (3) physical attributes, ma-
terial types and geometric properties of the scene. To un-
derstand stuff and things in context we introduce COCO-
Stuff, which augments 120,000 images of the COCO dataset
with pixel-wise annotations for 91 stuff classes. We introduce an efficient stuff
annotation protocol based on superpixels which leverages
the original thing annotations. We quantify the speed versus
quality trade-off of our protocol and explore the relation be-
tween annotation time and boundary complexity. Further-
more, we use COCO-Stuff to analyze: (a) the importance of
stuff and thing classes in terms of their surface cover and
how frequently they are mentioned in image captions; (b)
the spatial relations between stuff and things, highlighting
the rich contextual relations that make our dataset unique;
(c) the performance of a modern semantic segmentation
method on stuff and thing classes, and whether stuff is
easier to segment than things.
View details
Preview abstract
We introduce Fluid Annotation, an intuitive human-machine collaboration interface for annotating the class label and outline of every object and background region in an image. Fluid Annotation starts from the output of a strong neural network model, which the annotator can edit by correcting the labels of existing regions, adding new regions to cover missing objects, and removing incorrect regions. Fluid annotation has several attractive properties: (a) it is very efficient in terms of human annotation time; (b) it supports full images annotation in a single pass, as opposed to performing a series of small tasks in isolation, such as indicating the presence of objects, clicking on instances, or segmenting a single object known to be present. Fluid Annotation subsumes all these tasks in one unified interface. (c) it empowers the annotator to choose what to annotate and in which order. This enables to put human effort only on the errors the machine made, which helps using the annotation budget effectively. Through extensive experiments on the COCO+Stuff dataset, we demonstrate that Fluid Annotation leads to accurate annotations very efficiently, taking three times less annotation time than the popular LabelMe interface.
View details
Preview abstract
Manually annotating object bounding boxes is central to building computer vision datasets, and it is very time consuming (annotating ILSVRC [53] took 35s for one high-quality box [62]). It involves clicking on imaginary corners of a tight box around the object. This is difficult as these corners are often outside the actual object and several adjustments are required to obtain a tight box. We propose extreme clicking instead: we ask the annotator to click on four physical points on the object: the top, bottom, left- and right-most points. This task is more natural and these points are easy to find. We crowd-source extreme point annotations for PASCAL VOC 2007 and 2012 and show that (1) annotation time is only 7s per box, 5x faster than the traditional way of drawing boxes [62]; (2) the quality of the boxes is as good as the original ground-truth drawn the traditional way; (3) detectors trained on our annotations are as accurate as those trained on the original ground-truth. Moreover, our extreme clicking strategy not only yields box coordinates, but also four accurate boundary points. We show (4) how to incorporate them into GrabCut to obtain more accurate segmentations than those delivered when initializing it from bounding boxes; (5) semantic segmentations models trained on these segmentations outperform those trained on segmentations derived from bounding boxes.
View details
Preview abstract
Training object class detectors typically requires a large
set of images with objects annotated by bounding boxes.
However, manually drawing bounding boxes is very time
consuming. In this paper we greatly reduce annotation
time by proposing center-click annotations: we ask anno-
tators to click on the center of an imaginary bounding box
which tightly encloses the object instance. We then incor-
porate these clicks into existing Multiple Instance Learn-
ing techniques for weakly supervised object localization, to
jointly localize object bounding boxes over all training im-
ages. Extensive experiments on PASCAL VOC 2007 and
MS COCO show that: (1) our scheme delivers high-quality
detectors, performing substantially better than those pro-
duced by weakly supervised techniques, with a modest ex-
tra annotation effort; (2) these detectors in fact perform in a
range close to those trained from manually drawn bounding
boxes; (3) as the center-click task is very fast, our scheme
reduces total annotation time by 11× to 22×.
View details
The Devil is in the Decoders
Zbigniew Wojna
Vittorio Ferrari
Sergio Guadarrama
Nathan Silberman
Liang-chieh Chen
BMVC (2017)
Preview abstract
Many machine vision applications require predictions for every pixel of the input image (for example semantic segmentation, boundary detection). Models for such problems usually consist of encoders which decreases spatial resolution while learning a high-dimensional representation, followed by decoders who recover the original input resolution and result in low-dimensional predictions. While encoders have been studied rigorously, relatively few studies address the decoder side. Therefore this paper presents an extensive comparison of a variety of decoders for a variety of pixel-wise prediction tasks. Our contributions are: (1) Decoders matter: we observe significant variance in results between different types of decoders on various problems. (2) We introduce a novel decoder: bilinear additive upsampling. (3) We introduce new residual-like connections for decoders. (4) We identify two decoder types which give a consistently high performance.
View details
No Results Found
