PP-2013-08: Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model

PP-2013-08: Endriss, Ulle and Fernández, Raquel (2013) Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model. [Report]

[img]
Preview
Text (Full Text)
PP-2013-08.text.pdf

Download (258kB) | Preview
[img] Text (Abstract)
PP-2013-08.abstract.txt

Download (952B)

Abstract

Crowdsourcing, which offers new ways of cheaply and quickly gathering large amounts of information contributed by volunteers online, has revolutionised the collection of labelled data. Yet, to create annotated linguistic resources from this data, we face the challenge of having to combine the judgements of a potentially large group of annotators. In this paper we investigate how to aggregate individual annotations into a single collective annotation, taking inspiration from the field of social choice theory. We formulate a general formal model for collective annotation and propose several aggregation methods that go beyond the commonly used majority rule. We test some of our methods on data from a crowdsourcing experiment on textual entailment annotation.

Item Type: Report
Report Nr: PP-2013-08
Series Name: Prepublication (PP) Series
Year: 2013
Uncontrolled Keywords: computational linguistics; social choice theory
Subjects: Computation
Depositing User: Ulle Endriss
Date Deposited: 12 Oct 2016 14:37
Last Modified: 12 Oct 2016 14:37
URI: https://eprints.illc.uva.nl/id/eprint/480

Actions (login required)

View Item View Item