# TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models

arXiv: https://arxiv.org/abs/2309.01947 · HTML: https://mlsanity.com/paper/2309.01947/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [AlphaNet: Improved Training of Supernets with Alpha-Divergence](https://mlsanity.com/paper/2102.07954/) (2021)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [Improving RNN Transducer Based ASR with Auxiliary Tasks](https://mlsanity.com/paper/2011.03109/) (2020)
- [Dissecting User-Perceived Latency of On-Device E2E Speech Recognition](https://mlsanity.com/paper/2104.02207/) (2021)
- [Multi-Head State Space Model for Speech Recognition](https://mlsanity.com/paper/2305.12498/) (2023)
- [Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernet](https://mlsanity.com/paper/2110.08352/) (2021)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes](https://mlsanity.com/paper/2204.06164/) (2022)
- [Improving noise robustness of automatic speech recognition via parallel data and teacher-student learning](https://mlsanity.com/paper/1901.02348/) (2019)
- [Learning a Dual-Mode Speech Recognition Model via Self-Pruning](https://mlsanity.com/paper/2207.11906/) (2022)

## Built on by (most cited first, 4 of 4)

- [Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition](https://mlsanity.com/paper/2407.18930/) (2024)
- [Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression](https://mlsanity.com/paper/2501.18895/) (2025)
- [AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devices](https://mlsanity.com/paper/2511.22355/) (2025)
- [Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement](https://mlsanity.com/paper/2609.29867/) (2026)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) models need to be optimized for specific hardware before they can be deployed on devices. This can be done by tuning the model's hyperparameters or exploring variations in its architecture. Re-training and re-validating models after making these changes can be a resource-intensive task. This paper presents TODM (Train Once Deploy Many), a new approach to efficiently train many sizes of hardware-friendly on-device ASR models with comparable GPU-hours to that of a single training job. TODM leverages insights from prior work on Supernet, where Recurrent Neural Network Transducer (RNN-T) models share weights within a Supernet. It reduces layer sizes and widths of the Supernet to obtain subnetworks, making them smaller models suitable for all hardware types. We introduce a novel combination of three techniques to improve the outcomes of the TODM Supernet: adaptive dropouts, an in-place Alpha-divergence knowledge distillation, and the use of ScaledAdam optimizer. We validate our approach by comparing Supernet-trained versus individually tuned Multi-Head State Space Model (MH-SSM) RNN-T using LibriSpeech. Results demonstrate that our TODM Supernet either matches or surpasses the performance of manually tuned models by up to a relative of 3% better in word error rate (WER), while efficiently keeping the cost of training many models at a small constant.