mlsanityData to 7 Oct 2026

Position: Towards a Responsible LLM-empowered Multi-Agent Systems

arXiv 2502.01714 ↗First posted 3 Feb 2025cs.MA cs.AIcited by 7

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The rise of Agent AI and Large Language Model-powered Multi-Agent Systems (LLM-MAS) has underscored the need for responsible and dependable system operation. Tools like LangChain and Retrieval-Augmented Generation have expanded LLM capabilities, enabling deeper integration into MAS through enhanced knowledge retrieval and reasoning. However, these advancements introduce critical challenges: LLM agents exhibit inherent unpredictability, and uncertainties in their outputs can compound across interactions, threatening system stability. To address these risks, a human-centered design approach with active dynamic moderation is essential. Such an approach enhances traditional passive oversight by facilitating coherent inter-agent communication and effective system governance, allowing MAS to achieve desired outcomes more efficiently.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 50 of its 149 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

50 of 149 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Large Language Model Based Multi-Agent System Augmented Complex Event Processing Pipeline for Internet of Multimedia Things1 Jan
  2. 2024Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty12 Dec
  3. 2024The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap9 Dec
  4. 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI25 Oct
  5. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration18 Apr
  6. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?28 Feb
  7. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning8 Dec
  8. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration14 Nov
  9. 2023Multi-Agent Consensus Seeking via Large Language Models31 Oct
  10. 2023Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures5 Oct
  11. 2023Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models20 Jul
  12. 2022Improving alignment of dialogue agents via targeted human judgements28 Sep

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2018AI safety via debatecited by 384
  12. 2023Large Language Model Alignment: A Surveycited by 241
26 more
  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  4. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  5. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  6. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  7. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  8. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  9. 2024Moral Alignment for LLM Agentscited by 39
  10. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition Viewcited by 38
  11. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 35
  12. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 33
  13. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  14. 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
  15. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  16. 2022PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaborationcited by 23
  17. 2024Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysiscited by 21
  18. 2024SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similaritycited by 18
  19. 2024Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planningcited by 16
  20. 2024Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Plannerscited by 15
  21. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 13
  22. 2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
  23. 2024Understanding Knowledge Drift in LLMs through Misinformationcited by 9
  24. 2024Trust-Oriented Adaptive Guardrails for Large Language Modelscited by 8
  25. 2022Researching Alignment Research: Unsupervised Analysiscited by 7
  26. 2024LLM-guided Task and Motion Planning using Knowledge-based Reasoningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Thought Communication in Multiagent Collaborationcited by 28
  2. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  3. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  4. 2025Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothingcited by 9
  5. 2025Many LLMs Are More Utilitarian Than Onecited by 7
  6. 2025Why do AI agents communicate in human language?cited by 2
  7. 2025Multi-Stakeholder Alignment in LLM-Powered Collaborative AI Systems: A Multi-Agent Framework for Intelligent Tutoringcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-Enabled Multi-Agent Systems: Empirical Evaluation and Insights into Emerging Design Patterns & Paradigms
  2. 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems
  3. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment
  4. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  5. 2024Practical Considerations for Agentic LLM Systems
  6. 2025Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering
  7. 2025LLM-Powered AI Agent Systems and Their Applications in Industry
  8. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents