A Strategy for Evaluating Machine Translation Models of Different Paradigms

Resumen

Currently, translation models based on different paradigms coexist, which can yield mixed results with no clear hierarchy yet established. This poses a challenge for automatic evaluation metrics, which appear to be more reliable when comparing models within the same paradigm, but not when comparing models of very different types. Using a Spanish-Galician test suite of 335 sentences and three representative machine translation systems (from seq2seq to LLMs), we measure the correlation of eight automatic metrics with binary human correctness judgments and find that BLASER aligns best with human evaluation, though the correlation remains moderate for all metrics. To address the residual bias, we propose BLASERPPL, a perplexity-weighted extension of BLASER that modulates the semantic similarity score with perplexity-based weight derived from a target-language model. With a weight parameter alpha in [0.15, 0.20], the proposed metric recovers the human ranking for all three systems, rewarding translations that are both semantically faithful and linguistically natural without penalizing well-formed paraphrases, although not strictly literal.