# Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.15469 · HTML: https://mlsanity.com/paper/2403.15469/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Controlling the Output Length of Neural Machine Translation](https://mlsanity.com/paper/1910.10408/) (2019)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Isometric MT: Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08682/) (2021)

## Built on by (most cited first, 4 of 4)

- [Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling](https://mlsanity.com/paper/2602.00597/) (2026)
- [Length Aware Speech Translation for Video Dubbing](https://mlsanity.com/paper/2506.00740/) (2025)
- [From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimization](https://mlsanity.com/paper/2602.01068/) (2026)
- [CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization](https://mlsanity.com/paper/2603.16966/) (2026)

## Abstract (arXiv)

Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.