mlsanityData to 14 Oct 2026

ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management

arXiv 2607.20764 ↗First posted 22 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows. ARBIGRAPH represents each task as a natural-language problem with an executable Python solver, and composes tasks through typed intermediate states, instantiated here as scalar and list values. This design enables controllable task graphs whose length, dependency structure, distractor count, and value type can be varied while preserving exact automatic verification. We instantiate ARBIGRAPH with math, GSM-style word-problems, and Python-tracing task categories, and evaluate a Qwen3.5-27B tool-assisted agent across four topologies. The results show high accuracy on isolated tasks but substantial degradation on more complex dependent tasks: accuracy drops by up to 33.3% on branching chains of dependent math tasks. This shows that ARBIGRAPH exposes failures that are not visible from single-task evaluation alone. Our code, generated datasets, and evaluation results are available at https://github.com/pavelgolikov/ArbiGraph.git

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

33 of 40 references.

References · 33

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  12. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
21 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  2. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  3. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  4. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  5. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  6. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  7. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  8. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  9. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  10. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  11. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  12. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42
  13. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  14. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  15. 2025MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluationcited by 15
  16. 2026LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growthcited by 14
  17. 2025REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Oncecited by 12
  18. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  19. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  20. 2025Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structurescited by 5
  21. 2026Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalizationcited by 1

Built on by

Not cited yet.