Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate
Not on the map. Why?
Abstract (arXiv)
An assistant can stop repeating a deleted statement while its recurrent memory still carries that statement's influence. We examine this distinction by saving the state difference immediately after a record, carrying this saved difference, or receipt, through later updates, and comparing the corrected state with the state built from the same conversation with the record omitted. Unrolling the recurrence gives an exact criterion: transport reaches this never-stored state if and only if the additional differences created by later updates cancel after transport. We evaluate eleven conditions using 40 prospectively selected synthetic Kimi Linear contexts, with 40 separate calibration contexts and paired uncertainty estimates computed across complete records. Attention masking removed all 40 exact greedy target answers, yet a fixed three-query candidate-scoring attack with known candidate values and log-probability access attained AUC 0.740 (95% interval 0.695--0.809); the measured false-positive rate was 7.5%. Prompt-only forgetting still returned 33 of the 40 targets. Checkpoint replay matched the complete declared active state and audit logits in all 80 contexts, while preserving all 120 retained answers in the evaluation cohort. An independent Qwen cohort and matched original-bf16/8-bit Kimi controls confirmed native transport mismatch beyond the numerical error measured in matched controls. Replay also supported record replacement and successive deletions with a new record added between them. These results provide a practical audit of both the requested memory change and the assistant's remaining behavior, with replay work determined by the surviving suffix.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
24 of 30 references.
References · 24
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2019Machine Unlearningcited by 1,061
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
- 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
- 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
- 2024Titans: Learning to Memorize at Test Timecited by 328
12 more
- 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 261
- 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
- 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
- 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 84
- 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 63
- 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 42
- 2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
- 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
- 2026Models Designed to Forget: Machine Unlearning via Key Deletioncited by 3
- 2026Can an AI Assistant Really Forget? Auditable Deletion from Addressable Memorycited by 3
- 2026What a Deletion Certificate Covers, and Where It Expires: Auditable Removal from a Support-Vector Memorycited by 2
- 2025Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Modelscited by 1
Built on by
Not cited yet.