mlsanityData to 7 Oct 2026

RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

arXiv 2605.04523 ↗First posted 6 May 2026cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We present our winning system for Task~B (generation with reference passages) in SemEval-2026 Task~8: MTRAGEval. Our method is a heterogeneous ensemble of seven LLMs with two prompting variants, where a GPT-4o-mini judge selects the best candidate per instance. We ranked 1st out of 26 teams, achieving a conditioned harmonic mean of 0.7827 and outperforming the strongest baseline (gpt-oss-120b, 0.6390). Ablations show that diversity in model families, scales, and prompting strategies is essential, with the ensemble consistently beating any single model. We also introduce Meno-Lite-0.1, a 7B domain-adapted model with a strong cost--performance trade-off, and analyse MTRAGEval, highlighting annotation limitations and directions for improvement. Our code is publicly available: https://github.com/RaguTeam/ragu_mtrag_semeval

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

17 of 34 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  8. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  9. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  10. 2020Open Question Answering over Tables and Textcited by 163
  11. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  12. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
5 more
  1. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  2. 2025RAG-Anything: All-in-One RAG Frameworkcited by 18
  3. 2021NEREL: A Russian Dataset with Nested Named Entities, Relations and Eventscited by 13
  4. 2026MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversationscited by 8
  5. 2020RuREBus: a Case Study of Joint Named Entity Recognition and Relation Extraction from e-Government Domaincited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLMcited by 1
  2. 2026Little Brains, Big Feats: Exploring Compact Language Modelscited by 0
  3. 2026When Harness Beats Scale, and When Reading Beats Bothcited by 0