Jump to Content

Crowdsourcing a Gold Standard for Medical Relation Extraction with CrowdTruth

Anca Dumitrache
Lora Aroyo
Proceedings of the 2016 Collective Intelligence Conference
Google Scholar


In this paper, we make the following contributions: (1) a comparison of the quality and efficacy of annotations for medical relation extraction provided by both crowd and medical experts, showing that crowd annotations are equivalent to those of experts, with appropriate processing; (2) an openly available dataset of 900 English sentences for medical relation extraction, centering primarily on the cause relation, that have been processed with disagreement analysis and by experts.