# According to Me: Long-Term Personalized Referential Memory QA

arXiv: https://arxiv.org/abs/2603.01990 · HTML: https://mlsanity.com/paper/2603.01990/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 30 of 31)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](https://mlsanity.com/paper/2302.11713/) (2023)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval](https://mlsanity.com/paper/2406.04292/) (2024)
- [M+: Extending MemoryLLM with Scalable Long-Term Memory](https://mlsanity.com/paper/2502.00592/) (2025)
- [Disentangling Memory and Reasoning Ability in Large Language Models](https://mlsanity.com/paper/2411.13504/) (2024)
- [PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers](https://mlsanity.com/paper/2402.08327/) (2024)
- [PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering](https://mlsanity.com/paper/2402.16288/) (2024)
- [SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement](https://mlsanity.com/paper/2506.14035/) (2025)
- [Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://mlsanity.com/paper/2408.01866/) (2024)
- [OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering](https://mlsanity.com/paper/2409.08250/) (2024)

## Built on by (most cited first, 14 of 14)

- [MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory](https://mlsanity.com/paper/2605.15128/) (2026)
- [MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents](https://mlsanity.com/paper/2605.09530/) (2026)
- [Metis: Memory Foundation Model](https://mlsanity.com/paper/2607.26760/) (2026)
- [ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgetting](https://mlsanity.com/paper/2605.03804/) (2026)
- [Opal: Private Memory for Personal AI](https://mlsanity.com/paper/2604.02522/) (2026)
- [Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization](https://mlsanity.com/paper/2605.09996/) (2026)
- [Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering](https://mlsanity.com/paper/2606.16494/) (2026)
- [MobileMem: Learning from a Year of Mobile Experiences](https://mlsanity.com/paper/2608.13606/) (2026)
- [GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory](https://mlsanity.com/paper/2608.26983/) (2026)
- [Personal AI Agent for Camera Roll VQA](https://mlsanity.com/paper/2606.05275/) (2026)
- [MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale](https://mlsanity.com/paper/2608.02613/) (2026)
- [ICM-Bench: Person-Level Identity Reasoning in Multimodal Agents with Long-Term Memory](https://mlsanity.com/paper/2609.04438/) (2026)
- [Graph-Based Personalized Memory for LLM Agents: Representation, Evolution, Retrieval, and Evaluation](https://mlsanity.com/paper/2609.08599/) (2026)
- [Vela: User Behavior as Supervision Sparks LLM Intelligence](https://mlsanity.com/paper/2610.14709/) (2026)

## Abstract (arXiv)

Personalized AI assistants must recall and reason over long-term user memory, which naturally spans multiple modalities and sources such as images, videos, and emails. However, existing Long-term Memory benchmarks focus primarily on dialogue history, failing to capture realistic personalized references grounded in lived experience. We introduce ATM-Bench, the first benchmark for multimodal, multi-source personalized referential Memory QA. ATM-Bench contains approximately four years of privacy-preserving personal memory data and human-annotated question-answer pairs with ground-truth memory evidence, including queries that require resolving personal references, multi-evidence reasoning from multi-source and handling conflicting evidence. We propose Schema-Guided Memory (SGM) to structurally represent memory items originated from different sources. In experiments, we implement 5 state-of-the-art memory systems along with a standard RAG baseline and evaluate variants with different memory ingestion, retrieval, and answer generation techniques. We find poor performance (under 20% accuracy) on the ATM-Bench-Hard set, and that SGM improves performance over Descriptive Memory commonly adopted in prior works. Code available at: https://github.com/JingbiaoMei/ATM-Bench