PP-2014-11: Axiomatic Analysis of Aggregation Methods for Collective Annotation

PP-2014-11: Kruger, Justin and Endriss, Ulle and Fernandez, Raquel and Qing, Ciyang (2014) Axiomatic Analysis of Aggregation Methods for Collective Annotation. [Report]

Preview	Text (Full Text) PP-2014-11.text.pdf Download (320kB) | Preview
	Text (Abstract) PP-2014-11.abstract.txt Download (1kB)

Abstract

Crowdsourcing is an important tool, e.g., in computational
linguistics and computer vision, to effciently label large
amounts of data using nonexpert annotators. The individual
annotations collected need to be aggregated into a single
collective annotation. The hope is that the quality of
this collective annotation will be comparable to that of a
traditionally sourced expert annotation. In practice, most
scientists working with crowdsourcing methods use simple
majority voting to aggregate their data, although some have
also used probabilistic models and treated aggregation as a
problem of maximum likelihood estimation. The observation
that the aggregation step in a collective annotation exercise
may be considered a problem of social choice has only
been made very recently. Following up on this observation,
we show that the axiomatic method, as practiced in social
choice theory, can make a contribution to this important
domain and we develop an axiomatic framework for collective
annotation, focusing amongst other things on the notion of
an annotator's bias. We complement our theoretical study
with a discussion of a crowdsourcing experiment using data
from dialogue modelling in computational linguistics.

Item Type:	Report
Report Nr:	PP-2014-11
Series Name:	Prepublication (PP) Series
Year:	2014
Uncontrolled Keywords:	crowdsourcing; annotation; computational social choice
Subjects:	Computation
Depositing User:	Ulle Endriss
Date Deposited:	12 Oct 2016 14:37
Last Modified:	12 Oct 2016 14:37
URI:	https://eprints.illc.uva.nl/id/eprint/499

Actions (login required)

View Item