mlsanityData to 14 Oct 2026

Does Engram Do Memory Retrieval in Autoregressive Image Generation?

arXiv 2605.13179 ↗First posted 13 May 2026cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

The Engram module -- a hash-keyed, O(1) associative memory injected into Transformer layers -- was recently shown to improve large language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to vision with 2D spatial n-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-memory budget ratios ρ∈[0.17, 0.90], every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate -- inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire memory table frozen to N(0, 1) noise costs only DeltaFID=0.10 and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

16 of 19 references.

References · 16

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  7. 2016Pixel Recurrent Neural Networkscited by 816
  8. 2014Neural Turing Machinescited by 644
  9. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  10. 2014Memory Networkscited by 363
  11. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  12. 2024Randomized Autoregressive Visual Generationcited by 127
4 more
  1. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 98
  2. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 86
  3. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  4. 2025Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Modelcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Autoregressive Transformers Must Use Omega(n2 d) Memory
  2. 2025Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling
  3. 2026TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
  4. 2025Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
  5. 2025Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation
  6. 2026Knowledge Distillation for Visual Autoregressive Models
  7. 2025Grouped Speculative Decoding for Autoregressive Image Generation
  8. 2026Tiny-Engram: Trigger-Indexed Concept Tables for Generative Vision