mlsanityData to 14 Oct 2026

Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

arXiv 2610.03698 ↗First posted 2 Oct 2026cs.CVcited by 0

On the map: Mechanistic interpretability → Interpreting vision and multimodal models

Abstract (arXiv)

Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly understood. Recent architectures introduce dedicated register tokens to reduce high-norm out- lier patch tokens that emerge in background re- gions, yet the semantic and functional roles of both token types have not been fully established. In this paper, we analyze these roles by training sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2. Using an automated interpretability pipeline, UMAP clus- tering, and CLIP-space cross-checks, we find that register-token features are more strongly associ- ated with high-level semantic concepts. Outlier- token features, by contrast, are more often associ- ated with lower-level structural, background, and texture-dominant patterns. Causal ablations fur- ther reveal a substantial functional asymmetry: disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived fea- tures produces only a 0.31% drop. Together, our results provide evidence for token specialization in self-supervised ViTs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability. Builds on 9 of its 10 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

9 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Language Models Can Explain Visual Features via Steering23 Mar

Other references · 8

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2023Vision Transformers Need Registerscited by 871
  5. 2024Scaling and evaluating sparse autoencoderscited by 553
  6. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  7. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  8. 2025Vision Transformers Don't Need Trained Registerscited by 41

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decoding Register Tokens Preserves Global Scene Details
  2. 2026Test-Time Registers as Global Priors for Tokenized Image Generation
  3. 2026Do Vision Language Models Need to Process Image Tokens?
  4. 2026Registers Matter for Pixel-Space Diffusion Transformers
  5. 2024Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning
  6. 2026Taming Outlier Tokens in Diffusion Transformers
  7. 2025Causal Interpretation of Sparse Autoencoder Features in Vision
  8. 2026Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision