mlsanityData to 7 Oct 2026

AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts

arXiv 2601.20730 ↗First posted 28 Jan 2026cs.CLcited by 8

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce AgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 63 of its 79 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

63 of 79 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agents10 Jul
  2. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions7 Jul
  3. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents18 Jun
  4. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  5. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 58

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
38 more
  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  3. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  4. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  6. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  8. 2021Efficient Attentions for Long Document Summarizationcited by 342
  9. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  10. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  11. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  13. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  14. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  15. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  16. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  17. 2024Many-Shot In-Context Learningcited by 203
  18. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  19. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  20. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  21. 2025Qwen2.5-1M Technical Reportcited by 158
  22. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  23. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  24. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  25. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  26. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  27. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  28. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  29. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  30. 2025HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representationcited by 68
  31. 2025PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Pathscited by 60
  32. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  33. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  34. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  35. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  36. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  37. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  38. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 72
  2. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 17
  3. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  4. 2026AgentSwing: Adaptive Parallel Context Management Routing for Long-Horizon Web Agentscited by 9
  5. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  6. 2026Evaluating Memory Condensation Strategies for Coding Agents in Data-Driven Scientific Discoverycited by 1
  7. 2026World-Model Collapse as a Phase Transitioncited by 1
  8. 2026SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
  2. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
  3. 2026PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
  4. 2026Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents
  5. 2026AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents
  6. 2026MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems
  7. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents
  8. 2026SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent