Surprise as a Signal for Plasticity and Metacognition
Not on the map. Why?
Abstract (arXiv)
We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition. In the first system, a non-parametric episodic memory writes a new concept only when this surprise is high, and a periodic offline replay phase consolidates recent traces into a slow linear readout. On a continual stream of 1000 ImageNet classes with a frozen DINOv2 or I-JEPA backbone, the consolidation phase recovers 17.7 points of retention on the oldest classes for DINOv2 and 51.3 points for I-JEPA (single-seed runs), and an ablation shows that replaying only a recent window is worse than no replay at all. In few-shot evaluation the same memory reaches 91.6% on 5-way 1-shot mini-ImageNet, above a task-specific baseline, while a harder 500-way regime exposes the true difficulty. In the second system, the same surprise signal, computed in a shared text-image space, modulates the behaviour of a vision-language model: it answers assertively when a concept is known, hedges when it is partially familiar, and refuses to identify the object and asks for an explanation when it is novel, learning the concept from a single user utterance. The external detector separates known from novel concepts at an AUROC of 0.966 (95% CI +/-0.024), far above the model's own verbalised confidence (0.618), while its token-level confidence sits below chance under greedy decoding; after a sleep phase that empties the fast store, the system recalls 99.2% of fifty taught facts from the consolidated store while a base model recovers none. We report both systems as proof-of-concept, with explicit limitations, and position the second against recent episodic-memory and personalised-VLM work.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 30 Jun 2026.
Builds on
14 of 20 references.
References · 14
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
- 2025Qwen2.5-VL Technical Reportcited by 5,499
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
- 2017Prototypical Networks for Few-shot Learningcited by 3,011
- 2016Matching Networks for One Shot Learningcited by 2,251
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,354
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,058
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,018
- 2024Titans: Learning to Memorize at Test Timecited by 312
- 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models
- 2025Mitigating Catastrophic Forgetting and Mode Collapse in Text-to-Image Diffusion via Latent Replay
- 2026Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
- 2026Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning
- 2026Information as Structural Alignment: A Dynamical Theory of Continual Learning
- 2026Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
- 2026Amnesia: A Stealthy Replay Attack on Continual Learning Dreams
- 2026Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models