A Crowdsourced Frame Disambiguation Corpus with Ambiguity
Abstract
We present a resource for the task of FrameNetsemantic frame disambiguation over 5,000word-sentence pairs from the Wikipedia cor-pus.The annotations were collected usinga novel crowdsourcing approach with mul-tiple workers per sentence to captureinter-annotator disagreement.In contrast to thetypical approach of attributing the best singleframe to each word, we provide a list of frameswith disagreement-based scores that expressthe confidence with which each frame appliesto the word. This is based on the idea thatinter-annotator disagreement is at least partlycaused by ambiguity that is inherent to the textand frames. We have found many exampleswhere the semantics of individual frames over-lap sufficiently to make them acceptable alter-natives for interpreting a sentence. We haveargued that ignoring this ambiguity creates anoverly arbitrary target for training and eval-uating natural language processing systems -if humans cannot agree, why would we ex-pect the correct answer from a machine to beany different? To process this data we alsoutilized an expanded lemma-set provided bythe Framester system, which merges FN withWordNet to enhance coverage. Our datasetincludes annotations of 1,000 sentence-wordpairs whose lemmas are not part of FN. Finallywe present metrics for evaluating frame disam-biguation systems that account for ambiguity.
Research Areas
