mlsanityData to 14 Oct 2026

Substrate Asymmetry in User-Side Memory: A Diagnostic Framework

arXiv 2606.11712 ↗First posted 10 Jun 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

User-side memory in LLMs is typically scored as a single "personalization" capability: given a user's history, is the output more user-aware? We show this aggregate metric hides opposite-direction failures. Memory factorises into at least three orthogonal axes -- behavioral consistency (style, voice), factual presence (recall facts in history), and factual absence (abstain when a fact is absent) -- and no single substrate wins all three. Comparing per-user gamma-LoRA (a small LoRA adapter trained on each user's history; gamma denotes per-user, not per-task) against BGE-large dense top-K retrieval on a controlled 50-user synthetic corpus and a real-data probe (LaMP-3), we find gamma-LoRA decisively wins behavioral style while RAG decisively wins factual absence -- and the same query-projection cells in attention layers 21-35 causally load-bear both effects in opposite directions (zeroing those LoRA weights raises absence-probe TPR by +33 pp and drops presence-probe TPR by 20 pp). On the more heavily RLHF-tuned Llama-3.1-8B-Instruct the asymmetry strengthens, not heals: parametric memory's behavioral advantage collapses while its absence-calibration deficit against retrieval widens -- an alignment tax on parametric user-memory. On real-data LaMP-3, gamma-LoRA underperforms a majority baseline; a 9-condition mitigation sweep diagnoses this as instruction-following collapse, not substrate failure (a 9x2 cross-product shows the eval-time {1..5} logit mask drives main_acc to >=0.995 on every recipe), and the best training-time fix replicates bit-identically on Llama. Finally, substrate-selection routing is question-classification, not calibration: a 110M DistilBERT on the question text alone beats every logit-based router. We contribute the diagnostic framework, the diagnosed real-data negative, the alignment-tax replication, and the routing-as-classification finding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

26 of 34 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2016Deep Learning with Differential Privacycited by 2,728
  12. 2016Learning without Forgettingcited by 2,496
14 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  6. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  7. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  8. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  9. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  11. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  12. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  13. 2024PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Promptingcited by 14
  14. 2025UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answeringcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026User as Engram: Internalizing Per-User Memory as Local Parametric Edits
  2. 2025Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
  3. 2026MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
  4. 2026Latent Personal Memory: Represent personal memory as dynamic soft prompts
  5. 2026Defense-in-Depth for LLMs: Evaluating Memory Gates Against Activation-Induced and Memory-Induced Sycophancy
  6. 2026Bi-Mem: Bidirectional Construction of Hierarchical Memory for Personalized LLMs via Inductive-Reflective Agents
  7. 2026Evaluating Memory Structure in LLM Agents
  8. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents