# Adaptive Semantic Token Selection for AI-native Goal-oriented Communications

arXiv: https://arxiv.org/abs/2405.02330 · HTML: https://mlsanity.com/paper/2405.02330/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://mlsanity.com/paper/2004.09602/) (2020)
- [Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2105.15075/) (2021)
- [6G Networks: Beyond Shannon Towards Semantic and Goal-Oriented Communications](https://mlsanity.com/paper/2011.14844/) (2020)
- [Knowledge Distillation from Internal Representations](https://mlsanity.com/paper/1910.03723/) (2019)
- [Nonlinear Transform Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2112.10961/) (2021)
- [Deep Joint Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2211.08747/) (2022)
- [Goal-Oriented and Semantic Communication in 6G AI-Native Networks: The 6G-GOALS Approach](https://mlsanity.com/paper/2402.07573/) (2024)
- [Weight subcloning: direct initialization of transformers using larger pretrained ones](https://mlsanity.com/paper/2312.09299/) (2023)
- [Semantic Communication with Adaptive Universal Transformer](https://mlsanity.com/paper/2108.09119/) (2021)
- [Adaptive Resource Allocation for Semantic Communication Networks](https://mlsanity.com/paper/2312.01081/) (2023)
- [Conditional computation in neural networks: principles and research trends](https://mlsanity.com/paper/2403.07965/) (2024)
- [Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference](https://mlsanity.com/paper/2312.10193/) (2023)
- [Neural Networks Based Beam Codebooks: Learning mmWave Massive MIMO Beams that Adapt to Deployment and Hardware](https://mlsanity.com/paper/2006.14501/) (2020)

## Built on by (most cited first, 10 of 10)

- [Adaptive Semantic Token Communication for Transformer-based Edge Inference](https://mlsanity.com/paper/2505.17604/) (2025)
- [ToDMA: Large Model-Driven Massive Token Communications for Semantic Multiple Access](https://mlsanity.com/paper/2505.10946/) (2025)
- [Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge](https://mlsanity.com/paper/2509.09955/) (2025)
- [A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion](https://mlsanity.com/paper/2501.07451/) (2025)
- [Video TokenCom: Textual Intent-Guided Multi-Rate Video Token Communications with UEP-Based Adaptive Source-Channel Coding](https://mlsanity.com/paper/2603.02470/) (2026)
- [Training-Free Multi-User Generative Semantic Communications via Null-Space Diffusion Sampling](https://mlsanity.com/paper/2405.09866/) (2024)
- [Adaptive Pareto-Optimal Token Merging for Edge Transformer Models in Semantic Communication](https://mlsanity.com/paper/2509.09168/) (2025)
- [SAGE: Training-Free Semantic Evidence Composition for Edge-Cloud Inference under Hard Uplink Budgets](https://mlsanity.com/paper/2604.19623/) (2026)
- [Token-Oriented Semantic Communication with Pretrained Vision Transformers](https://mlsanity.com/paper/2608.25410/) (2026)
- [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel design for AI-native goal-oriented communications, exploiting transformer neural networks under dynamic inference constraints on bandwidth and computation. Transformers have become the standard architecture for pretraining large-scale vision and text models, and preliminary results have shown promising performance also in deep joint source-channel coding (JSCC). Here, we consider a dynamic model where communication happens over a channel with variable latency and bandwidth constraints. Leveraging recent works on conditional computation, we exploit the structure of the transformer blocks and the multihead attention operator to design a trainable semantic token selection mechanism that learns to select relevant tokens (e.g., image patches) from the input signal. This is done dynamically, on a per-input basis, with a rate that can be chosen as an additional input by the user. We show that our model improves over state-of-the-art token selection mechanisms, exhibiting high accuracy for a wide range of latency and bandwidth constraints, without the need for deploying multiple architectures tailored to each constraint. Last, but not least, the proposed token selection mechanism helps extract powerful semantics that are easy to understand and explain, paving the way for interpretable-by-design models for the next generation of AI-native communication systems.