PP-2013-14: Evaluating Long Range Reordering with Permutation-Forests

PP-2013-14: Stanojevic, Milos and Sima'an, Khalil (2013) Evaluating Long Range Reordering with Permutation-Forests. [Report]

Text (Full Text)

Download (453kB) | Preview
[img] Text (Abstract)

Download (969B)


Automatically evaluating the quality of word order of MT systems is challenging yet crucial for MT evaluation. Existing approaches em- ploy string-based metrics, which are computed over the permutations of word positions in system output relative to a reference transla- tion. We introduce a new metric computed over Permutation Forests (PEFs), tree-based representations that decompose permutations re- cursively. Relative to string-based metrics, PEFs offer advantages for evaluating long range reordering. We compare the present PEFs metric against five known reordering metrics on WMT13 data for ten language pairs. The PEFs metric shows better correlation with human ranking than the other metrics almost on all language pairs. None of the other metrics exhibits as stable behavior across language pairs.

Item Type: Report
Report Nr: PP-2013-14
Series Name: Prepublication (PP) Series
Year: 2013
Uncontrolled Keywords: Machine Translation; Evaluation; Reordering
Depositing User: Khalil Sima'an
Date Deposited: 12 Oct 2016 14:37
Last Modified: 12 Oct 2016 14:37
URI: https://eprints.illc.uva.nl/id/eprint/486

Actions (login required)

View Item View Item