# Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroup

arXiv: https://arxiv.org/abs/2312.05795 · HTML: https://mlsanity.com/paper/2312.05795/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [Layer-adaptive sparsity for the Magnitude-based Pruning](https://mlsanity.com/paper/2010.07611/) (2020)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference](https://mlsanity.com/paper/2307.02628/) (2023)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning](https://mlsanity.com/paper/2009.08065/) (2020)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [Structured Compression by Weight Encryption for Unstructured Pruning and Quantization](https://mlsanity.com/paper/1905.10138/) (2019)
- [OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation](https://mlsanity.com/paper/2310.07749/) (2023)
- [Large Multi-modal Encoders for Recommendation](https://mlsanity.com/paper/2310.20343/) (2023)

## Built on by (most cited first, 8 of 8)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.08519/) (2024)
- [Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework](https://mlsanity.com/paper/2507.04636/) (2025)
- [CASP: Compression of Large Multimodal Models Based on Attention Sparsity](https://mlsanity.com/paper/2503.05936/) (2025)
- [ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task](https://mlsanity.com/paper/2503.04444/) (2025)
- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)
- [Learning a Single Token to Replace Long System Prompts in LLMs](https://mlsanity.com/paper/2511.23271/) (2025)
- [Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning](https://mlsanity.com/paper/2606.11853/) (2026)

## Abstract (arXiv)

The deployment of Large Multimodal Models (LMMs) within AntGroup has significantly advanced multimodal tasks in payment, security, and advertising, notably enhancing advertisement audition tasks in Alipay. However, the deployment of such sizable models introduces challenges, particularly in increased latency and carbon emissions, which are antithetical to the ideals of Green AI. This paper introduces a novel multi-stage compression strategy for our proprietary LLM, AntGMM. Our methodology pivots on three main aspects: employing small training sample sizes, addressing multi-level redundancy through multi-stage pruning, and introducing an advanced distillation loss design. In our research, we constructed a dataset, the Multimodal Advertisement Audition Dataset (MAAD), from real-world scenarios within Alipay, and conducted experiments to validate the reliability of our proposed strategy. Furthermore, the effectiveness of our strategy is evident in its operational success in Alipay's real-world multimodal advertisement audition for three months from September 2023. Notably, our approach achieved a substantial reduction in latency, decreasing it from 700ms to 90ms, while maintaining online performance with only a slight performance decrease. Moreover, our compressed model is estimated to reduce electricity consumption by approximately 75 million kWh annually compared to the direct deployment of AntGMM, demonstrating our commitment to green AI initiatives. We will publicly release our code and the MAAD dataset after some reviewsfootnotehttps://github.com/MorinW/AntGMM_Pruning.