PP-2013-14: Stanojevic, Milos and Sima'an, Khalil (2013) Evaluating Long Range Reordering with Permutation-Forests. [Report]
Preview |
Text (Full Text)
PP-2013-14.text.pdf Download (453kB) | Preview |
![[thumbnail of Abstract]](https://eprints.illc.uva.nl/style/images/fileicons/text.png) |
Text (Abstract)
PP-2013-14.abstract.txt Download (969B) |
Abstract
Automatically evaluating the quality of word order of MT systems is
challenging yet crucial for MT evaluation. Existing approaches em-
ploy string-based metrics, which are computed over the permutations
of word positions in system output relative to a reference transla-
tion. We introduce a new metric computed over Permutation Forests
(PEFs), tree-based representations that decompose permutations re-
cursively. Relative to string-based metrics, PEFs offer advantages
for evaluating long range reordering. We compare the present PEFs
metric against five known reordering metrics on WMT13 data for
ten language pairs. The PEFs metric shows better correlation with
human ranking than the other metrics almost on all language pairs.
None of the other metrics exhibits as stable behavior across language
pairs.
| Item Type: | Report |
|---|---|
| Report Nr: | PP-2013-14 |
| Series Name: | Prepublication (PP) Series |
| Year: | 2013 |
| Uncontrolled Keywords: | Machine Translation; Evaluation; Reordering |
| Depositing User: | Khalil Sima'an |
| Date Deposited: | 12 Oct 2016 14:37 |
| Last Modified: | 12 Oct 2016 14:37 |
| URI: | https://eprints.illc.uva.nl/id/eprint/486 |
Actions (login required)
 |
View Item |
