# FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference

arXiv: https://arxiv.org/abs/2601.13143 · HTML: https://mlsanity.com/paper/2601.13143/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving.
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Placed in

- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://mlsanity.com/paper/2306.09093/) (2023)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [ImageBind-LLM: Multi-modality Instruction Tuning](https://mlsanity.com/paper/2309.03905/) (2023)
- [Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference](https://mlsanity.com/paper/2405.05803/) (2024)
- [video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2406.15704/) (2024)
- [LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference](https://mlsanity.com/paper/2407.14057/) (2024)
- [ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst](https://mlsanity.com/paper/2305.16103/) (2023)
- [Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs](https://mlsanity.com/paper/2310.08915/) (2023)
- [AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models](https://mlsanity.com/paper/2410.18325/) (2024)
- [CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2403.04640/) (2024)
- [Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs](https://mlsanity.com/paper/2404.10308/) (2024)
- [AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2505.20862/) (2025)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)

## Built on by (most cited first, 7 of 7)

- [OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2605.12056/) (2026)
- [Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning](https://mlsanity.com/paper/2606.10533/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs](https://mlsanity.com/paper/2607.25669/) (2026)
- [Allocation Before Ranking: Decoupled Token Compression for OmniLLMs](https://mlsanity.com/paper/2608.01665/) (2026)
- [OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2609.37052/) (2026)
- [Audio Token Attention Is Predictable Before the Language Model Runs](https://mlsanity.com/paper/2609.38878/) (2026)

## Abstract (arXiv)

In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.