# Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

arXiv: https://arxiv.org/abs/2605.30087 · HTML: https://mlsanity.com/paper/2605.30087/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 46 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 8 of 8)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios](https://mlsanity.com/paper/2505.16944/) (2025)
- [LifeBench: A Benchmark for Long-Horizon Multi-Source Memory](https://mlsanity.com/paper/2603.03781/) (2026)
- [Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method](https://mlsanity.com/paper/2604.11209/) (2026)
- [CLASH: A Benchmark for Cross-Modal Contradiction Detection](https://mlsanity.com/paper/2511.19199/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict](https://mlsanity.com/paper/2608.13921/) (2026)

## Abstract (arXiv)

Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method's conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse.