Woodpecker - A Linguistic Diagnostic Evaluation Platform for Machine Translation Systems

The Woodpecker is a platform to diagnose the linguistic translation capability of machine translation (MT) systems. Instead of assigning a general score to an MT system, Woodpecker diagnoses the system capability of handling various important linguistic test cases called checkpoints. A checkpoint is a linguistically motivated unit (e.g. an ambiguous word, a noun phrase, a verb-object collocation, a prepositional phrase etc.) that is predefined in a linguistic taxonomy for diagnostic evaluation. The reference of a checkpoint is its corresponding part in the target sentence. Based on syntax parse trees and word alignments over source-testing and target-reference data, Woodpecker can extract various checkpoints automatically from source and target sentences and linguistically evaluate the translation performance of an MT system by computing the matching degree between its translation result and the references of checkpoints.