# Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2609.39134 · HTML: https://mlsanity.com/paper/2609.39134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 26 of 26)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models](https://mlsanity.com/paper/2403.15388/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [TokenPacker: Efficient Visual Projector for Multimodal LLM](https://mlsanity.com/paper/2407.02392/) (2024)
- [Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs](https://mlsanity.com/paper/2412.01818/) (2024)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models](https://mlsanity.com/paper/2412.00447/) (2024)
- [AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models](https://mlsanity.com/paper/2410.05346/) (2024)
- [LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence](https://mlsanity.com/paper/2605.25979/) (2026)
- [Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models](https://mlsanity.com/paper/2404.10335/) (2024)
- [FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models](https://mlsanity.com/paper/2505.19536/) (2025)
- [Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models](https://mlsanity.com/paper/2410.06699/) (2024)
- [Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models](https://mlsanity.com/paper/2407.21659/) (2024)
- [VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models](https://mlsanity.com/paper/2505.17440/) (2025)
- [On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression](https://mlsanity.com/paper/2601.21531/) (2026)
- [Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector](https://mlsanity.com/paper/2508.13739/) (2025)
- [EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling](https://mlsanity.com/paper/2604.17087/) (2026)
- [Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models](https://mlsanity.com/paper/2601.12042/) (2026)

## Abstract (arXiv)

Visual-token compression improves the efficiency of large vision-language models, but can expose failures that full-token evaluation misses. We study adversarial images that preserve full-token correctness yet induce errors after compression, even when both inference paths succeed on the clean image. Creating such failures is challenging because perturbing token importance can also damage the visual content needed for full-token inference. We propose Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens. In the primary LLaVA-1.5-7B setting, FATA uses only vision-encoder gradients, without access to the deployed compressor, token budget, or downstream task. Across four visually dependent task subsets and four compressors under a controlled reconstruction protocol, FATA achieves SR = 96.3% full-token accuracy retention and CBR = 22.1% conditional blinding, compared with 89.8% and 15.7% for CAA. Ablations support the role of both objectives in balancing compressed-path failure against full-token preservation. FATA also has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate. These findings motivate assessing adversarial robustness jointly across full-token and compressed inference.