PP-2013-08: Endriss, Ulle and Fernández, Raquel (2013) Collective Annotation of Linguistic Resources: Basic Principles and a Formal Model. [Report]
Preview |
Text (Full Text)
PP-2013-08.text.pdf Download (258kB) | Preview |
![[thumbnail of Abstract]](https://eprints.illc.uva.nl/style/images/fileicons/text.png) |
Text (Abstract)
PP-2013-08.abstract.txt Download (952B) |
Abstract
Crowdsourcing, which offers new ways of cheaply and quickly gathering
large amounts of information contributed by volunteers online, has
revolutionised the collection of labelled data. Yet, to create
annotated linguistic resources from this data, we face the challenge
of having to combine the judgements of a potentially large group of
annotators. In this paper we investigate how to aggregate individual
annotations into a single collective annotation, taking inspiration
from the field of social choice theory. We formulate a general formal
model for collective annotation and propose several aggregation
methods that go beyond the commonly used majority rule. We test some
of our methods on data from a crowdsourcing experiment on textual
entailment annotation.
| Item Type: | Report |
|---|---|
| Report Nr: | PP-2013-08 |
| Series Name: | Prepublication (PP) Series |
| Year: | 2013 |
| Uncontrolled Keywords: | computational linguistics; social choice theory |
| Subjects: | Computation |
| Depositing User: | Ulle Endriss |
| Date Deposited: | 12 Oct 2016 14:37 |
| Last Modified: | 12 Oct 2016 14:37 |
| URI: | https://eprints.illc.uva.nl/id/eprint/480 |
Actions (login required)
 |
View Item |
