# Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends

arXiv: https://arxiv.org/abs/2409.14457 · HTML: https://mlsanity.com/paper/2409.14457/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 160 of its 214 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Builds on (most cited first, 30 of 160)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)

## Built on by (most cited first, 26 of 26)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Internet of Agents: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2505.07176/) (2025)
- [Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents](https://mlsanity.com/paper/2412.18371/) (2024)
- [From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration](https://mlsanity.com/paper/2603.04474/) (2026)
- [LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networks](https://mlsanity.com/paper/2504.14556/) (2025)
- [The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerability](https://mlsanity.com/paper/2510.18563/) (2025)
- [Position: Towards a Responsible LLM-empowered Multi-Agent Systems](https://mlsanity.com/paper/2502.01714/) (2025)
- [LLM Applications: Current Paradigms and the Next Frontier](https://mlsanity.com/paper/2503.04596/) (2025)
- [ComAgent: Multi-LLM based Agentic AI Empowered Intelligent Wireless Networks](https://mlsanity.com/paper/2601.19607/) (2026)
- [When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents](https://mlsanity.com/paper/2607.01641/) (2026)
- [FRSICL: LLM-Enabled In-Context Learning Flight Resource Allocation for Fresh Data Collection in UAV-Assisted Wildfire Monitoring](https://mlsanity.com/paper/2507.10134/) (2025)
- [ScamAgents: How AI Agents Can Simulate Human-Level Scam Calls](https://mlsanity.com/paper/2508.06457/) (2025)
- [Exploring Agentic Artificial Intelligence Systems: Towards a Typological Framework](https://mlsanity.com/paper/2508.00844/) (2025)
- [Graph Representation-based Model Poisoning on the Heterogeneous Internet of Agents](https://mlsanity.com/paper/2511.07176/) (2025)
- [Interpretable Anomaly-Based DDoS Detection in AI-RAN with XAI and LLMs](https://mlsanity.com/paper/2507.21193/) (2025)
- [Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark](https://mlsanity.com/paper/2510.02356/) (2025)
- [Enabling Regulatory Multi-Agent Collaboration: Architecture, Challenges, and Solutions](https://mlsanity.com/paper/2509.09215/) (2025)
- [AIC-VDS: Attention-Based In-Context Learning for Joint Velocity Control and Data Collection Scheduling in Multi-UAV-Assisted Pipeline Monitoring](https://mlsanity.com/paper/2510.05698/) (2025)
- [Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs](https://mlsanity.com/paper/2603.17902/) (2026)
- [Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs](https://mlsanity.com/paper/2605.07961/) (2026)
- [Security of OpenClaw Agents: Fundamentals, Attacks, and Countermeasures](https://mlsanity.com/paper/2605.25435/) (2026)
- [FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification](https://mlsanity.com/paper/2607.28225/) (2026)
- [Multi-Agent Systems: From Classical Paradigms to Large Foundation Model-Enabled Futures](https://mlsanity.com/paper/2604.18133/) (2026)
- [Agentic MPC for Semantic Control System Resynthesis](https://mlsanity.com/paper/2606.12774/) (2026)
- [SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs](https://mlsanity.com/paper/2609.36879/) (2026)

## Abstract (arXiv)

With the rapid advancement of large models (LMs), the development of general-purpose intelligent agents powered by LMs has become a reality. It is foreseeable that in the near future, LM-driven general AI agents will serve as essential tools in production tasks, capable of autonomous communication and collaboration without human intervention. This paper investigates scenarios involving the autonomous collaboration of future LM agents. We review the current state of LM agents, the key technologies enabling LM agent collaboration, and the security and privacy challenges they face during cooperative operations. To this end, we first explore the foundational principles of LM agents, including their general architecture, key components, enabling technologies, and modern applications. We then discuss practical collaboration paradigms from data, computation, and knowledge perspectives to achieve connected intelligence among LM agents. After that, we analyze the security vulnerabilities and privacy risks associated with LM agents, particularly in multi-agent settings, examining underlying mechanisms and reviewing current and potential countermeasures. Lastly, we propose future research directions for building robust and secure LM agent ecosystems.