mlsanityData to 14 Oct 2026

DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

arXiv 2608.20664 ↗First posted 21 Aug 2026cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

DreamBench-SWE is a multi-session benchmark for software-agent memory hygiene in which later software tasks depend on non-inferable evidence from earlier sessions and are scored by executable hidden oracles. We report the original scaled v2 fold and a separately preregistered v2.1 successor audit designed after that study but frozen before successor outcome inspection. The successor run completed 360/360 work units and 720/720 S3 cells across four conditions. In the original fold, the primary DF-hybrid--B5 contrast was null (95/180 versus 89/180; clustered p=.518, Holm p=1), not evidence of equivalence, and C9/C10 retained B0-headroom limitations. In the successor, no external memory achieved 21/180 passes (rate 0.1167), deterministic verbatim event memory 82/180 (rate 0.4556), the typed-plus-raw reference probe 83/180 (rate 0.4611), and one pinned hosted Mem0 literal-storage configuration 97/180 (rate 0.5389). The registered six-slot Family A retained unavailable slots at p=1; all three available comparisons against no memory rejected after Holm correction. Both preregistered mechanism contrasts were unavailable after pre-evaluation conformance rejection. The secondary literal-storage-versus-verbatim comparison was nonconfirmatory and sensitivity-dependent, while the comparison with the reference probe did not reject. The audit therefore supports DreamBench-SWE as a discriminating executable profile benchmark and characterizes one exact hosted-memory configuration, but it does not establish an external-system mechanism, superiority among memory-bearing conditions, equivalence, or broad product generality. The original v2.0.5 findings and artifacts remain unchanged.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

32 of 45 references.

References · 32

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  8. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2023Cognitive Architectures for Language Agentscited by 429
  11. 2024Agent Workflow Memorycited by 383
  12. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
20 more
  1. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  2. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  3. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  4. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 81
  5. 2025A Self-Improving Coding Agentcited by 79
  6. 2025Tongyi DeepResearch Technical Reportcited by 66
  7. 2025Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMscited by 60
  8. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  9. 2025Chain-of-Retrieval Augmented Generationcited by 49
  10. 2025Sleep-time Compute: Beyond Inference Scaling at Test-timecited by 43
  11. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 32
  12. 2026Useful Memories Become Faulty When Continuously Updated by LLMscited by 20
  13. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  14. 2026MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Taskscited by 15
  15. 2026Are We Ready For An Agent-Native Memory System?cited by 15
  16. 2026Structurally Aligned Subtask-Level Memory for Software Engineering Agentscited by 12
  17. 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agentscited by 12
  18. 2026WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interactioncited by 7
  19. 2026TRUSTMEM: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memorycited by 6
  20. 2026Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvementcited by 5

Built on by

Not cited yet.