Gal-SummEval: Meta-Evaluation of Abstractive Summarization in Galician

Resumen

This work introduces Gal-SummEval, the first expert-curated benchmark for Galician automatic text summarization, comprising manually annotated summaries across five criteria: coherence, consistency, fluency, relevance, and 5W1H (covering the core questions who, what, where, when, why, and how). We use this resource for a dual-methodology study: benchmarking traditional metrics and LLM-as-a-judge models against human judgments, and exploring cross-lingual transfer by fine-tuning evaluator models on Spanish and Basque. Our results reveal that no single model evaluator excels across all dimensions. While LLM-based judges effectively capture 5W1H and fluency, traditional metrics remain superior for detecting consistency and relevance. Furthermore, cross-lingual transfer proves effective for overcoming resource scarcity, though fine-tuned models still fail to match the correlation levels of traditional metrics. Ultimately, traditional metrics exhibit higher overall performance.