# Localizing and Mitigating Memorization in Image Autoregressive Models

arXiv: https://arxiv.org/abs/2509.00488 · HTML: https://mlsanity.com/paper/2509.00488/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [Randomized Autoregressive Visual Generation](https://mlsanity.com/paper/2411.00776/) (2024)
- [Can Neural Network Memorization Be Localized?](https://mlsanity.com/paper/2307.09542/) (2023)
- [Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models](https://mlsanity.com/paper/2406.02366/) (2024)
- [Localizing Paragraph Memorization in Language Models](https://mlsanity.com/paper/2403.19851/) (2024)
- [Memorization in Self-Supervised Learning Improves Downstream Generalization](https://mlsanity.com/paper/2401.12233/) (2024)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)
- [Privacy Attacks on Image AutoRegressive Models](https://mlsanity.com/paper/2502.02514/) (2025)
- [Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted](https://mlsanity.com/paper/2406.18566/) (2024)
- [Localizing Memorization in SSL Vision Encoders](https://mlsanity.com/paper/2409.19069/) (2024)
- [Captured by Captions: On Memorization and its Mitigation in CLIP Models](https://mlsanity.com/paper/2502.07830/) (2025)

## Built on by (most cited first, 1 of 1)

- [NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models](https://mlsanity.com/paper/2605.30393/) (2026)

## Abstract (arXiv)

Image AutoRegressive (IAR) models have achieved state-of-the-art performance in speed and quality of generated images. However, they also raise concerns about memorization of their training data and its implications for privacy. This work explores where and how such memorization occurs within different image autoregressive architectures by measuring a fine-grained memorization. The analysis reveals that memorization patterns differ across various architectures of IARs. In hierarchical per-resolution architectures, it tends to emerge early and deepen with resolutions, while in IARs with standard autoregressive per token prediction, it concentrates in later processing stages. These localization of memorization patterns are further connected to IARs' ability to memorize and leak training data. By intervening on their most memorizing components, we significantly reduce the capacity for data extraction from IARs with minimal impact on the quality of generated images. These findings offer new insights into the internal behavior of image generative models and point toward practical strategies for mitigating privacy risks.