PP-2014-12: Qing, Ciyang and Endriss, Ulle and Fernandez, Raquel and Kruger, Justin (2014) Empirical Analysis of Aggregation Methods for Collective Annotation. [Report]
Preview |
Text (Full Text)
PP-2014-12.text.pdf Download (275kB) | Preview |
![[thumbnail of Abstract]](https://eprints.illc.uva.nl/style/images/fileicons/text.png) |
Text (Abstract)
PP-2014-12.abstract.txt Download (534B) |
Abstract
We investigate methods for aggregating the judgements
of multiple individuals in a linguistic annotation task into
a collective judgement. We define several aggregators
that take the reliability of annotators into account and
thus go beyond the commonly used majority vote, and
we empirically analyse their performance on new datasets
of crowdsourced data.
| Item Type: | Report |
|---|---|
| Report Nr: | PP-2014-12 |
| Series Name: | Prepublication (PP) Series |
| Year: | 2014 |
| Uncontrolled Keywords: | crowdsourcing; annotation |
| Depositing User: | Ulle Endriss |
| Date Deposited: | 12 Oct 2016 14:37 |
| Last Modified: | 12 Oct 2016 14:37 |
| URI: | https://eprints.illc.uva.nl/id/eprint/500 |
Actions (login required)
 |
View Item |
