mlsanityData to 14 Oct 2026

MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration

arXiv 2503.15272 ↗First posted 19 Mar 2025cs.CL cs.AIcited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Multi-agent collaboration among models has shown promise in reasoning tasks but is underexplored in long-form generation tasks like summarization and question-answering. We extend multi-agent multi-model reasoning to generation, specifically to improving faithfulness through refinement, i.e., revising model-generated outputs to remove factual inconsistencies. We investigate how iterative collaboration among multiple instances and types of large language models (LLMs) enhances subtasks in the refinement process, such as error detection, critiquing unfaithful sentences, and making corrections based on critiques. We design intrinsic evaluations for each subtask, with our findings indicating that both multi-agent (multiple instances) and multi-model (diverse LLM types) approaches benefit error detection and critiquing. Additionally, reframing critiquing and refinement as reranking rather than generation tasks improves multi-agent performance. We consolidate these insights into a final "recipe" called Multi-Agent Multi-Model Refinement (MAMM-Refine), where multi-agent and multi-model collaboration significantly boosts performance on three summarization datasets as well as on long-form question answering, demonstrating the effectiveness and generalizability of our recipe.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 56 of its 71 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

56 of 71 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Improving LLM Reasoning with Multi-Agent Tree-of-Thought Validator Agent17 Sep
  2. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
  3. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration11 Jul
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 52

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  9. 2023Teaching Large Language Models to Self-Debugcited by 988
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
38 more
  1. 2019ELI5: Long Form Question Answeringcited by 575
  2. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  3. 2023How Language Model Hallucinations Can Snowballcited by 311
  4. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  6. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  7. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  8. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  9. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  10. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  11. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  12. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  13. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  14. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  15. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  16. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  17. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  18. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  19. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  20. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  21. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  22. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 84
  23. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
  24. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  25. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 68
  26. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  27. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  28. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  29. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  30. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48
  31. 2023Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articlescited by 45
  32. 2020Evidence-based Factual Error Correctioncited by 44
  33. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  34. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  35. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  36. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  37. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Frameworkcited by 19
  38. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedbackcited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  2. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
  3. 2025Collab-REC: An LLM-based Agentic Framework for Balancing Recommendations in Tourismcited by 7
  4. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  5. 2025Learning from Self Critique and Refinement for Faithful LLM Summarizationcited by 1
  6. 2025PaceLLM: Brain-Inspired Large Language Models for Long-Context Understandingcited by 0
  7. 2025MetaSynth: Multi-Agent Metadata Generation from Implicit Feedback in Black-Box Systemscited by 0
  8. 2026A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarizationcited by 0
  9. 2026CLEAR: Causal Context-Based Agentic Reasoning for Vulnerability Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
  2. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
  3. 2025Self-Correcting Large Language Models: Generation vs. Multiple Choice
  4. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoning
  5. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklists
  6. 2025Adaptive Multi-Agent Response Refinement in Conversational Systems
  7. 2026Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
  8. 2025Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table Reasoning