Who said what: Modeling individual labelers improves classification
Abstract
Data are often labelled by many different experts with each expert only labeling a small fraction of the data and each data point being labelled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning mixing proportions for combining them in sample-specific ways. This allows us to give more weight to more reliable experts and makes it possible to take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improved computer-aided diagnosis of diabetic retinopathy, where the experts are human doctors and the data are retinal images. We compare our method against those of Welinder and Perona, and Mnih and Hinton. Our work offers an innovative approach for dealing with the myriad real-world settings that lack ground truth labels.
Research Areas
