mlsanityData to 14 Oct 2026

Mechanistically Interpreting Compression in Vision-Language Models

arXiv 2603.25035 ↗First posted 26 Mar 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Compressed vision-language models (VLMs) are widely used to reduce memory and compute costs, making them a suitable choice for real-world deployment. However, compressing these models raises concerns about whether internal computations and safety behaviors are preserved. In this work, we use causal circuit analysis and crosscoder-based feature comparisons to examine how pruning and quantization fundamentally change the internals across representative VLMs. We observe that pruning generally keeps circuit structure intact but rotates and attenuates internal features, while quantization modifies the circuits at a higher level yet leaves the surviving features better aligned. Leveraging this insight, we also introduce VLMSafe-420, a novel benchmark that pairs harmful inputs with matched benign counterfactuals across various safety categories. Our findings show that pruning causes a sharp drop in genuine refusal behavior, suggesting that the choice of compression has safety implications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

21 of 30 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  8. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  10. 2023Localizing Model Behavior with Path Patchingcited by 195
  11. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  12. 2024Finding Transformer Circuits with Edge Pruningcited by 60
9 more
  1. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  2. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  3. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
  4. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  5. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 23
  6. 2024Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularitycited by 14
  7. 2025SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretabilitycited by 9
  8. 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 8
  9. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models
  2. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption
  3. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
  4. 2025Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
  5. 2026Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
  6. 2025Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study
  7. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models
  8. 2026When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models