# ReplayLens: Auditing Agents' Use of Outcomes

arXiv: https://arxiv.org/abs/2609.34177 · HTML: https://mlsanity.com/paper/2609.34177/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM agents and tool use

## Builds on (most cited first, 15 of 15)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Counterfactual Credit Assignment in Model-Free Reinforcement Learning](https://mlsanity.com/paper/2011.09464/) (2020)
- [Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures](https://mlsanity.com/paper/2606.08275/) (2026)
- [PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents](https://mlsanity.com/paper/2608.04003/) (2026)
- [REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces](https://mlsanity.com/paper/2606.09071/) (2026)
- [Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay](https://mlsanity.com/paper/2608.19760/) (2026)
- [MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recovery](https://mlsanity.com/paper/2606.24595/) (2026)
- [AttriMem: Attribution-Guided Process Feedback for Agent Memory Construction](https://mlsanity.com/paper/2607.21106/) (2026)
- [HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents](https://mlsanity.com/paper/2606.16285/) (2026)
- [MemFail: Stress-Testing Failure Modes of LLM Memory Systems](https://mlsanity.com/paper/2605.26667/) (2026)
- [Chain-of-Experience for Continual LLM Improvement](https://mlsanity.com/paper/2608.18027/) (2026)
- [Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit](https://mlsanity.com/paper/2608.29605/) (2026)

## Abstract (arXiv)

When an agent reuses logged experience, a changed decision may reflect the recorded score, the action's name, or the record's position in storage. Standard memory evaluations do not reveal which relationship drives that change. We introduce ReplayLens, a black-box audit that changes one relationship in the stored history at a time, holds the remaining interface fixed, and measures the resulting decision. Four interventions target four relationships. Outcome reassignment swaps which scores belong to which actions. Pair transport moves intact action-score pairs to new record slots. Consistent renaming relabels actions in both history and menu. Key-slot reassignment changes both score attachment and position. A constructive separation shows why the audit is needed: two memory writers with identical endpoint accuracy respond differently to the same replay, so conventional evaluation cannot resolve the underlying dependence. On black-box LLM interfaces, swapping scores changes decisions while moving intact pairs does not, separating score attachment from record order. A bounded-memory study exposes ingestion-order sensitivity that endpoint comparison misses. In sequential experiment planning, altered historical scores redirect exploration and reduce final utility despite fresh measurements. A code-debugging agent with sealed hidden tests shows the same pattern outside model selection. ReplayLens provides a relationship-level audit for deciding whether logged experience can be merged, reordered, or reindexed safely.