mlsanityData to 7 Oct 2026

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

arXiv 2610.00406 ↗First posted 30 Sep 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation also fits: RAG robustness and adaptive retrieval

Abstract (arXiv)

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 6 of its 13 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

6 of 13 references.

References · 6

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  6. 2026RALS: Resources and Baselines for Romanian Automatic Lexical Simplificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems
  2. 2024GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering
  3. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
  4. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  5. 2024VERA: Validation and Evaluation of Retrieval-Augmented Systems
  6. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  7. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
  8. 2025GRILE: A Benchmark for Grammar Reasoning and Explanation in Romanian LLMs