Performant Lightweight Encoders for Spanish in the Legal and Administrative Domains

Resumen

This work introduces two lightweight, domain-adapted models for Spanish legal and administrative information retrieval, developed within the ALIA initiative: a bi-encoder and a cross-encoder built on the MrBERT-es backbone. Training combines a synthetic data generation pipeline grounded in legal-administrative sources with a six-phase curriculum of progressively increasing difficulty and positive-aware hard-negative mining. Evaluation on general-domain and domain-specific MTEB benchmarks shows that the bi-encoder achieves the best results on legal-administrative retrieval, outperforming larger general-purpose models such as bge-m3 and Qwen3-Embedding. The cross-encoder remains competitive with substantially larger rerankers on specialized Spanish legal datasets. These findings demonstrate that targeted domain adaptation and careful training design can compensate for model scale in specialized retrieval.