mlsanityData to 14 Oct 2026

AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learning

arXiv 2609.05435 ↗First posted 30 Jun 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Can language agents continually learn from experience, turning earlier interactions into reusable capabilities? AhaBench evaluates this ability through exploration after solved hidden-state puzzles, computational transfer after mathematical teaching, and sustained business operation under delayed feedback. The benchmark is agnostic to how an agent learns; the evaluated agents use fixed model weights. Curriculum profiles, teaching contrasts, and daily trajectories reveal a common challenge: using explicit guidance is more reliable than generalizing beyond it or sustaining useful behavior. Across the Puzzle panel, the advantage over matched cold targets is 36.0-53.5 points greater with trace support than at the trace-free endpoint; Qwen 3.6 Plus nevertheless retains a +12.57-point post-curriculum gain. In Euler, worked procedures yield 80.0-100.0% held-out accuracy across models, while question-plus-answer teaching yields 0.0-73.9%. Vending trajectories separate sustained profit, late recovery, and incomplete operation: Doubao Seed 2.0 Pro finishes nominal operation at +495 but averages -10 over the year. Together, these results make continual learning an operational target: experience should yield capabilities that remain effective as guidance, inputs, and business states change. We release tasks, validators, a simulator, records, and analyses for developing agents that turn useful insights into lasting abilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

54 of 62 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2016Matching Networks for One Shot Learningcited by 2,255
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  15. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  16. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  17. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  18. 2016Progressive Neural Networkscited by 1,361
  19. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  20. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  21. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  22. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  23. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  24. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  25. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  26. 2018Experience Replay for Continual Learningcited by 1,095
  27. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  28. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  29. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  30. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  31. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  32. 2022Transformers learn in-context by gradient descentcited by 732
  33. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  34. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  35. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  36. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  37. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  38. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.