# Accelerating Transducers through Adjacent Token Merging

arXiv: https://arxiv.org/abs/2306.16009 · HTML: https://mlsanity.com/paper/2306.16009/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Placed in

- Efficient LLM inference and serving / [Audio and omni-modal token compression](https://mlsanity.com/inference/audio-and-omni-modal-token-compression/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition](https://mlsanity.com/paper/2109.01163/) (2021)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition](https://mlsanity.com/paper/2008.05750/) (2020)
- [Variable-rate hierarchical CPC leads to acoustic unit discovery in speech](https://mlsanity.com/paper/2206.02211/) (2022)
- [Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation](https://mlsanity.com/paper/2102.07935/) (2021)
- [On-demand compute reduction with stochastic wav2vec 2.0](https://mlsanity.com/paper/2204.11934/) (2022)
- [Improving RNN-T ASR Accuracy Using Context Audio](https://mlsanity.com/paper/2011.10538/) (2020)
- [On Compressing Sequences for Self-Supervised Speech Models](https://mlsanity.com/paper/2210.07189/) (2022)
- [Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition](https://mlsanity.com/paper/2011.00771/) (2020)

## Built on by (most cited first, 15 of 15)

- [A Survey of Token Compression for Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2507.20198/) (2025)
- [OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models](https://mlsanity.com/paper/2511.14582/) (2025)
- [KV Shifting Attention Enhances Language Modeling](https://mlsanity.com/paper/2411.19574/) (2024)
- [AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression](https://mlsanity.com/paper/2602.21233/) (2026)
- [DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression](https://mlsanity.com/paper/2603.15685/) (2026)
- [FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model](https://mlsanity.com/paper/2410.03007/) (2024)
- [Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models](https://mlsanity.com/paper/2604.06871/) (2026)
- [OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2605.18041/) (2026)
- [OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2605.12056/) (2026)
- [Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey](https://mlsanity.com/paper/2510.00078/) (2025)
- [CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation](https://mlsanity.com/paper/2410.05146/) (2024)
- [OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models](https://mlsanity.com/paper/2607.23193/) (2026)
- [OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs](https://mlsanity.com/paper/2607.25669/) (2026)
- [Token Merging for Multilingual Speech Recognition: A Systematic Study Across Model Scale and Fine-Tuning](https://mlsanity.com/paper/2609.13151/) (2026)
- [AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs](https://mlsanity.com/paper/2610.13600/) (2026)

## Abstract (arXiv)

Recent end-to-end automatic speech recognition (ASR) systems often utilize a Transformer-based acoustic encoder that generates embedding at a high frame rate. However, this design is inefficient, particularly for long speech signals due to the quadratic computation of self-attention. To address this, we propose a new method, Adjacent Token Merging (A-ToMe), which gradually combines adjacent tokens with high similarity scores between their key values. In this way, the total time step could be reduced, and the inference of both the encoder and joint network is accelerated. Experiments on LibriSpeech show that our method can reduce 57% of tokens and improve the inference speed on GPU by 70% without any notable loss of accuracy. Additionally, we demonstrate that A-ToMe is also an effective solution to reduce tokens in long-form ASR, where the input speech consists of multiple utterances.