# Dynamic Transformer for Efficient Machine Translation on Embedded Devices

arXiv: https://arxiv.org/abs/2107.08199 · HTML: https://mlsanity.com/paper/2107.08199/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 17 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Dynamic Sparsity Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2005.10627/) (2020)
- [Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms](https://mlsanity.com/paper/2105.03596/) (2021)
- [Optimising Resource Management for Embedded Machine Learning](https://mlsanity.com/paper/2105.03608/) (2021)
- [Scalable Transformers for Neural Machine Translation](https://mlsanity.com/paper/2106.02242/) (2021)

## Built on by (most cited first, 4 of 4)

- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices](https://mlsanity.com/paper/2401.08965/) (2024)
- [LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models](https://mlsanity.com/paper/2605.17653/) (2026)

## Abstract (arXiv)

The Transformer architecture is widely used for machine translation tasks. However, its resource-intensive nature makes it challenging to implement on constrained embedded devices, particularly where available hardware resources can vary at run-time. We propose a dynamic machine translation model that scales the Transformer architecture based on the available resources at any particular time. The proposed approach, 'Dynamic-HAT', uses a HAT SuperTransformer as the backbone to search for SubTransformers with different accuracy-latency trade-offs at design time. The optimal SubTransformers are sampled from the SuperTransformer at run-time, depending on latency constraints. The Dynamic-HAT is tested on the Jetson Nano and the approach uses inherited SubTransformers sampled directly from the SuperTransformer with a switching time of <1s. Using inherited SubTransformers results in a BLEU score loss of <1.5% because the SubTransformer configuration is not retrained from scratch after sampling. However, to recover this loss in performance, the dimensions of the design space can be reduced to tailor it to a family of target hardware. The new reduced design space results in a BLEU score increase of approximately 1% for sub-optimal models from the original design space, with a wide range for performance scaling between 0.356s - 1.526s for the GPU and 2.9s - 7.31s for the CPU.