S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
Not on the map. Why?
Abstract (arXiv)
Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene--Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene--event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor--support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score--token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches (0.48) F1 and (0.40) BLEU with (1,073) evidence tokens per question, about (15.8×) fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only (189)tokens.On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 10 Apr 2026.
Builds on
25 of 47 references.
References · 25
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
13 more
- 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
- 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
- 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
- 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 180
- 2019Interactive Fiction Games: A Colossal Adventurecited by 168
- 2017Neural Episodic Controlcited by 151
- 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
- 2016Model-Free Episodic Controlcited by 102
- 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 102
- 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
- 2026EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agentscited by 9
Built on by
Citing papers per year: 2026: 2.
2 papers.