mlsanityData to 7 Oct 2026

LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration

arXiv 2402.11550 ↗First posted 18 Feb 2024cs.CL cs.AIcited by 38

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive performance in understanding language and executing complex reasoning tasks. However, LLMs with long context windows have been notorious for their expensive training costs and high inference latency. Even the most advanced models such as GPT-4 and Claude2 often make mistakes when processing inputs of over 100k tokens, a phenomenon also known as textitlost in the middle. In this paper, we propose textscLongAgent, a method based on multi-agent collaboration, which scales LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority in long-text processing compared to GPT-4. In textscLongAgent, a leader is responsible for understanding user intent and directing team members to acquire information from documents. Due to members' hallucinations, it is non-trivial for a leader to obtain accurate information from the responses of dozens to hundreds of members. To address this, we develop an textitinter-member communication mechanism to resolve response conflicts caused by hallucinations through information sharing. Our experimental results indicate that textscLongAgent offers a promising alternative for long-text processing. The agent team instantiated with LLaMA-7B achieves significant improvements in tasks such as 128k-long text retrieval, multi-hop question answering, compared to GPT-4.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 34 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

17 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration11 Jul

Other references · 16

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2023Large Language Models as Tool Makerscited by 293
  10. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  11. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  12. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
4 more
  1. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  2. 2024Long Context Compression with Activation Beaconcited by 101
  3. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  4. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 24; 2026: 5.

38 papers.

Sort:

By area:

  1. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  2. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  3. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  5. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  6. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  7. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
  8. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  9. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  10. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  11. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  12. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
26 more
  1. 2024LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Modelscited by 17
  2. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 13
  3. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  4. 2024Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understandingcited by 9
  5. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  6. 2025When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Frameworkcited by 8
  7. 2026MetaGen: Self-Evolving Roles and Topologies for Multi-Agent LLM Reasoningcited by 8
  8. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  9. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  10. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
  11. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  12. 2025SwarmSys: Decentralized Swarm-Inspired Agents for Scalable and Adaptive Reasoningcited by 5
  13. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  14. 2025SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Controlcited by 4
  15. 2025The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemascited by 4
  16. 2025GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memorycited by 4
  17. 2025APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUscited by 3
  18. 2025BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answeringcited by 3
  19. 2025Multi-Agent Retrieval-Augmented Framework for Evidence-Based Counterspeech Against Health Misinformationcited by 3
  20. 2025Cochain: Balancing Insufficient and Excessive Collaboration in LLM Agent Workflowscited by 2
  21. 2025LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Textcited by 2
  22. 2025Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarizationcited by 1
  23. 2024Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Rolescited by 0
  24. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 0
  25. 2026GOPAgen: Codec-Aware Agentic Long-Video Understanding with Structured Memorycited by 0
  26. 2026DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Collaboration in Multi-Agent Systems
  2. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  3. 2025Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning
  4. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents
  5. 2026Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Bias
  6. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models
  7. 2026LSTM-MAS: A Long Short-Term Memory Inspired Multi-Agent System for Long-Context Understanding
  8. 2026Beyond tokens: a unified framework for latent communication in LLM-based multi-agent systems