mlsanityData to 14 Oct 2026

Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)

arXiv 2603.26743 ↗First posted 23 Mar 2026cs.CV cs.AI cs.LGcited by 0

On the map: Mechanistic interpretability → Interpreting vision and multimodal models

Abstract (arXiv)

Dynamic head pruning in Vision Transformers (ViTs) improves efficiency by removing redundant attention heads, but existing pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic pruning, leveraging their ability to disentangle dense embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual embedding of the ViT and amplify the sparse latents with different strategies to alter pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% to 82%) while reducing head usage (0.72 to 0.33) via heads h2 and h5. These results show that sparse latent features enable class-specific control of dynamic pruning, effectively bridging pruning efficiency and mechanistic interpretability in ViTs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability. Builds on 3 of its 4 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

3 of 4 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptation6 Dec

Other references · 2

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  2. 2013k-Sparse Autoencoderscited by 215

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
  2. 2025HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer
  3. 2023Bridging The Gaps Between Token Pruning and Full Pre-training via Masked Fine-tuning
  4. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
  5. 2026SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable
  6. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  7. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  8. 2025Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering