# Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge

arXiv: https://arxiv.org/abs/2509.09955 · HTML: https://mlsanity.com/paper/2509.09955/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [A Vision of 6G Wireless Systems: Applications, Trends, Technologies, and Open Research Problems](https://mlsanity.com/paper/1902.10265/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models](https://mlsanity.com/paper/2409.10197/) (2024)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [SwinJSCC: Taming Swin Transformer for Deep Joint Source-Channel Coding](https://mlsanity.com/paper/2308.09361/) (2023)
- [AdvFlow: Inconspicuous Black-box Adversarial Attacks using Normalizing Flows](https://mlsanity.com/paper/2007.07435/) (2020)
- [Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention](https://mlsanity.com/paper/2209.13802/) (2022)
- [Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications](https://mlsanity.com/paper/2502.12096/) (2025)

## Built on by (most cited first, 7 of 7)

- [ZACH-ViT: A Zero-Token Vision Transformer with ShuffleStrides Data Augmentation for Robust Lung Ultrasound Classification](https://mlsanity.com/paper/2510.17650/) (2025)
- [Ada-TokenCom: Rate-Adaptive Token Communications via Large-Model-Driven Token Compression and Generation](https://mlsanity.com/paper/2608.28086/) (2026)
- [Topology-Preserving Deep Joint Source-Channel Coding for Semantic Communication](https://mlsanity.com/paper/2603.17126/) (2026)
- [Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling](https://mlsanity.com/paper/2609.01232/) (2026)
- [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026)
- [Queue-Aware Token Communications: Bridging AI Representation and Wireless Service](https://mlsanity.com/paper/2610.16730/) (2026)
- [Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression](https://mlsanity.com/paper/2610.18039/) (2026)

## Abstract (arXiv)

Large-scale transformers are central to modern semantic communication, yet their high computational and communication costs hinder deployment on resource-constrained edge devices. This paper introduces a training-free framework for adaptive token merging, a novel mechanism that compresses transformer representations at runtime by selectively merging semantically redundant tokens under per-layer similarity thresholds. Unlike prior fixed-ratio reduction, our approach couples merging directly to input redundancy, enabling data-dependent adaptation that balances efficiency and task relevance without retraining. We cast the discovery of merging strategies as a multi-objective optimization problem and leverage Bayesian optimization to obtain Pareto-optimal trade-offs between accuracy, inference cost, and communication cost. On ImageNet classification, we match the accuracy of the unmodified transformer with 30% fewer floating-point operations per second and under 20% of the original communication cost, while for visual question answering our method achieves performance competitive with the full LLaVA model at less than one-third of the compute and one-tenth of the bandwidth. Finally, we show that our adaptive merging is robust across varying channel conditions and provides inherent privacy benefits, substantially degrading the efficacy of model inversion attacks. Our framework provides a practical and versatile solution for deploying powerful transformer models in resource-limited edge intelligence scenarios.