mlsanityData to 14 Oct 2026

Accelerating Transducers through Adjacent Token Merging

arXiv 2306.16009 ↗First posted 28 Jun 2023cs.CL eess.AS eess.SPcited by 15

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

Recent end-to-end automatic speech recognition (ASR) systems often utilize a Transformer-based acoustic encoder that generates embedding at a high frame rate. However, this design is inefficient, particularly for long speech signals due to the quadratic computation of self-attention. To address this, we propose a new method, Adjacent Token Merging (A-ToMe), which gradually combines adjacent tokens with high similarity scores between their key values. In this way, the total time step could be reduced, and the inference of both the encoder and joint network is accelerated. Experiments on LibriSpeech show that our method can reduce 57% of tokens and improve the inference speed on GPU by 70% without any notable loss of accuracy. Additionally, we demonstrate that A-ToMe is also an effective solution to reduce tokens in long-form ASR, where the input speech consists of multiple utterances.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 37 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

30 of 37 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2022Token Merging: Your ViT But Fastercited by 1,021
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  7. 2015Listen, Attend and Spellcited by 543
  8. 2015Attention-Based Models for Speech Recognitioncited by 401
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  10. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  11. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  12. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
18 more
  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  2. 2021Learned Token Pruning for Transformerscited by 139
  3. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  4. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  5. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  6. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  7. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  8. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  9. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  10. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  11. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  12. 2020Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognitioncited by 23
  13. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  14. 2021Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillationcited by 12
  15. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10
  16. 2020Improving RNN-T ASR Accuracy Using Context Audiocited by 8
  17. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 8
  18. 2020Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  2. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 22
  3. 2024KV Shifting Attention Enhances Language Modelingcited by 8
  4. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 6
  5. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  6. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  7. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Modelscited by 4
  8. 2026OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Modelscited by 4
  9. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 3
  10. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  11. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  12. 2026OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Modelscited by 1
3 more
  1. 2026OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMscited by 1
  2. 2026Token Merging for Multilingual Speech Recognition: A Systematic Study Across Model Scale and Fine-Tuningcited by 0
  3. 2026AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MURMUR: An Efficient Inference System for Long-Form ASR
  2. 2022Improving Streaming End-to-End ASR on Transformer-based Causal Models with Encoder States Revision Strategies
  3. 2026Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference
  4. 2022Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames
  5. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognition
  6. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation
  7. 2025Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models
  8. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper