mlsanityData to 14 Oct 2026

Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification

arXiv 2608.17247 ↗First posted 18 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a current task. We use this setting to develop an empirical audit protocol for structured intermediate outputs: first audit dataset shortcuts, then isolate bundled prompt changes, check whether intermediate labels are answer-associated, test decomposed semantic evidence, and audit provider-level execution failures. A 480-example synthetic development set initially suggested large gains from a state-structured prompt bundle, but TF-IDF diagnostics showed lexical separability and no positive standalone Ignore cases. We therefore construct a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies. On this set, exposing the four state definitions improves accuracy, but an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts policy predictions, but because those labels deterministically map to policies, this is a label-conditioning diagnostic rather than evidence of a faithful internal mechanism. Family-level and seed-stability analyses further show that example-level accuracy overstates counterfactual consistency: complete four-way family success is rare. An exploratory follow-up that elicits decomposed semantic evidence also fails to improve routing for the cleanly evaluated endpoint; the corresponding GPT-OSS condition was unavailable because of provider-side request validation. We evaluate policy classification only, not downstream responses, tool actions, or memory-store mutation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

39 of 46 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  12. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
27 more
  1. 2022Mass-Editing Memory in a Transformercited by 1,001
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  4. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  6. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  7. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
  8. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  9. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
  10. 2025Walk the Talk? Measuring the Faithfulness of Large Language Model Explanationscited by 45
  11. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 37
  12. 2026Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agentscited by 32
  13. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  14. 2026MemTX: Transactional Belief Commit for Stateful Agent Memorycited by 15
  15. 2026The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agentscited by 10
  16. 2026MAP-Graph: Provenance-Aware Shared Memory for Multi-Agent Workflowscited by 9
  17. 2026How Early Is Early Enough? Design-Dependent Observation-Window Sufficiency in Subscription Churn Predictioncited by 8
  18. 2026From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agentscited by 8
  19. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  20. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 7
  21. 2026DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderationcited by 4
  22. 2026Class Weighting versus Amount Conditioning in Credit-Card Fraud Detection: A Dollar-Metric Study with a Temporal Explanation Auditcited by 4
  23. 2026S-Path-RAG: Semantic-Aware Shortest-Path Retrieval Augmented Generation for Multi-Hop Knowledge Graph Question Answeringcited by 3
  24. 2026Interpretable vs Learned Encoders for High-Cardinality Fraud Detectioncited by 3
  25. 2026Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Accesscited by 3
  26. 2026RegDivergence-101: An LLM Benchmark for Cross-Jurisdiction Regulatory Contradiction Detection in Life Sciencescited by 2
  27. 2026Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrievalcited by 1

Built on by

Not cited yet.