mlsanityData to 7 Oct 2026

M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation

arXiv 2412.20127 ↗First posted 28 Dec 2024cs.CL cs.AIcited by 13

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at https://github.com/SU-JIAYUAN/M-MAD.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 24 of its 41 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

24 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 23

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  12. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
11 more
  1. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  3. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  4. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  5. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  6. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  7. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  8. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  9. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  10. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  11. 2024From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2026Agent-as-a-Judgecited by 15
  2. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  3. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  4. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  5. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  6. 2026GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translationcited by 2
  7. 2025CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debatecited by 1
  8. 2025RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Designcited by 1
  9. 2025Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotationscited by 1
  10. 2026ReCCur: A Recursive Corner-Case Curation Framework for Robust Vision-Language Understanding in Open and Edge Scenarioscited by 0
  11. 2026AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimizationcited by 0
  12. 2026Source-Free MT Evaluation Is Not MT Evaluationcited by 0
1 more
  1. 2026EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpretingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation
  2. 2025MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation
  3. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation
  4. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  5. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  6. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications
  7. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  8. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results