Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
As a model-agnostic approach to long context modeling, multi-agent systems can process inputs longer than a large language model's context window without retraining or architectural modifications. However, their performance often heavily relies on hand-crafted multi-agent collaboration strategies and prompt engineering, which limit generalizability. In this work, we introduce a principled framework that formalizes the model-agnostic long context modeling problem as a compression problem, yielding an information-theoretic compression objective. Building on this framework, we propose Graph of Agents (GoA), which dynamically constructs an input-dependent collaboration structure that maximizes this objective. For Llama 3.1 8B and Qwen3 8B across six document question answering benchmarks, GoA improves the average F1 score of retrieval-augmented generation by 5.7% and a strong multi-agent baseline using a fixed collaboration structure by 16.35%, respectively. Even with only a 2K context window, GoA surpasses the 128K context window Llama 3.1 8B on LongBench, showing a dramatic increase in effective context length. Our source code is available at https://github.com/tjoo512/graph-of-agents.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 37 of its 42 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
37 of 42 references.
Closest ancestors · 1
Same direction, newest first
Other references · 36
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
24 more
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
- 2017The NarrativeQA Reading Comprehension Challengecited by 597
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
- 2019On Variational Bounds of Mutual Informationcited by 559
- 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
- 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
- 2022Exploring Length Generalization in Large Language Modelscited by 206
- 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
- 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
- 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
- 2020Unsupervised Question Decomposition for Question Answeringcited by 118
- 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
- 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 66
- 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
- 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
- 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 43
- 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
- 2025Long-Context Generalization with Sparse Attentioncited by 21
- 2024LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environmentscited by 18
- 2025ReAgent: Reversible Multi-Agent Reasoning for Knowledge-Enhanced Multi-Hop QAcited by 13
- 2024Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generationcited by 6
- 2023Deep Clustering with a Constraint for Topological Invariance based on Symmetric InfoNCEcited by 5
- 2025BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answeringcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration
- 2026Coding Agents are Effective Long-Context Processors
- 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
- 2025Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning
- 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
- 2026Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices
- 2026MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation
- 2026Can Small Agents Collaborate to Beat a Single Large Language Model?