PP-2014-11: Axiomatic Analysis of Aggregation Methods for Collective Annotation

PP-2014-11: Kruger, Justin and Endriss, Ulle and Fernandez, Raquel and Qing, Ciyang (2014) Axiomatic Analysis of Aggregation Methods for Collective Annotation. [Report]

Text (Full Text)

Download (320kB) | Preview
[img] Text (Abstract)

Download (1kB)


Crowdsourcing is an important tool, e.g., in computational linguistics and computer vision, to effciently label large amounts of data using nonexpert annotators. The individual annotations collected need to be aggregated into a single collective annotation. The hope is that the quality of this collective annotation will be comparable to that of a traditionally sourced expert annotation. In practice, most scientists working with crowdsourcing methods use simple majority voting to aggregate their data, although some have also used probabilistic models and treated aggregation as a problem of maximum likelihood estimation. The observation that the aggregation step in a collective annotation exercise may be considered a problem of social choice has only been made very recently. Following up on this observation, we show that the axiomatic method, as practiced in social choice theory, can make a contribution to this important domain and we develop an axiomatic framework for collective annotation, focusing amongst other things on the notion of an annotator's bias. We complement our theoretical study with a discussion of a crowdsourcing experiment using data from dialogue modelling in computational linguistics.

Item Type: Report
Report Nr: PP-2014-11
Series Name: Prepublication (PP) Series
Year: 2014
Uncontrolled Keywords: crowdsourcing; annotation; computational social choice
Subjects: Computation
Depositing User: Ulle Endriss
Date Deposited: 12 Oct 2016 14:37
Last Modified: 12 Oct 2016 14:37
URI: https://eprints.illc.uva.nl/id/eprint/499

Actions (login required)

View Item View Item