# Test-Time Registers as Global Priors for Tokenized Image Generation

arXiv: https://arxiv.org/abs/2607.16824 · HTML: https://mlsanity.com/paper/2607.16824/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 36 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [See What You Are Told: Visual Attention Sink in Large Multimodal Models](https://mlsanity.com/paper/2503.03321/) (2025)
- [Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens](https://mlsanity.com/paper/2501.07730/) (2025)
- [Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation](https://mlsanity.com/paper/2111.04318/) (2021)
- [LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report](https://mlsanity.com/paper/2405.00732/) (2024)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [Language-Guided Image Tokenization for Generation](https://mlsanity.com/paper/2412.05796/) (2024)
- [Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations](https://mlsanity.com/paper/2403.07241/) (2024)
- [AToken: A Unified Tokenizer for Vision](https://mlsanity.com/paper/2509.14476/) (2025)
- [Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2404.02845/) (2024)
- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [Uncovering Memorization Effect in the Presence of Spurious Correlations](https://mlsanity.com/paper/2501.00961/) (2025)
- [To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models](https://mlsanity.com/paper/2510.08510/) (2025)
- [CoMA: Compositional Human Motion Generation with Multi-modal Agents](https://mlsanity.com/paper/2412.07320/) (2024)
- [Highly Compressed Tokenizer Can Generate Without Training](https://mlsanity.com/paper/2506.08257/) (2025)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models](https://mlsanity.com/paper/2506.20629/) (2025)
- [Attention (as Discrete-Time Markov) Chains](https://mlsanity.com/paper/2507.17657/) (2025)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)

## Abstract (arXiv)

Attention-based models often develop attention sinks, where a small number of tokens repeatedly attract attention and accumulate unusually large activations. In vision transformers, these outliers are closely related to registers, which have been diagnostically linked to global, low-frequency image structure. Existing work has largely studied registers through interpretability analyses and linear probes, leaving open whether they can be operationalized as plug-and-play signals for generation without retraining. We revisit this question in tokenized image generation. Using OpenCLIP and DINOv2 on ImageNet, we find that test-time register features exhibit stronger low-frequency concentration than both [CLS] readouts and patch-mean features, and show a consistent (albeit moderate) correlation with pixel-space DCT low-frequency energy. Motivated by these diagnostics, we introduce RegToken, a training-free procedure that converts register structure into a small set of global prior tokens by (i) NFN-based layer localization, (ii) TokenRank-guided subspace extraction, and (iii) a projection-and-conservation update on the register subspace. Inserted into a frozen compact 1D token generation pipeline, RegToken improves ImageNet generation and alignment metrics (e.g., FID-5k 20.5 to 20.1, SigLIP 3.6 to 3.9) without modifying pretrained weights, and accelerates test-time optimization (Steps@τ 74 to 52). Overall, our results suggest that structures often viewed as attention artifacts can be repurposed as lightweight global priors for tokenized generation.