PP-2014-12: Empirical Analysis of Aggregation Methods for Collective Annotation

PP-2014-12: Qing, Ciyang and Endriss, Ulle and Fernandez, Raquel and Kruger, Justin (2014) Empirical Analysis of Aggregation Methods for Collective Annotation. [Report]

[thumbnail of Full Text]
Text (Full Text)

Download (275kB) | Preview
[thumbnail of Abstract] Text (Abstract)

Download (534B)


We investigate methods for aggregating the judgements
of multiple individuals in a linguistic annotation task into
a collective judgement. We define several aggregators
that take the reliability of annotators into account and
thus go beyond the commonly used majority vote, and
we empirically analyse their performance on new datasets
of crowdsourced data.

Item Type: Report
Report Nr: PP-2014-12
Series Name: Prepublication (PP) Series
Year: 2014
Uncontrolled Keywords: crowdsourcing; annotation
Depositing User: Ulle Endriss
Date Deposited: 12 Oct 2016 14:37
Last Modified: 12 Oct 2016 14:37
URI: https://eprints.illc.uva.nl/id/eprint/500

Actions (login required)

View Item View Item