# CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation

arXiv: https://arxiv.org/abs/2506.19952 · HTML: https://mlsanity.com/paper/2506.19952/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 12 of 12)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Extremely low-resource machine translation for closely related languages](https://mlsanity.com/paper/2105.13065/) (2021)
- [Continual Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2212.09097/) (2022)

## Built on by (most cited first, 2 of 2)

- [Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation](https://mlsanity.com/paper/2602.09691/) (2026)
- [Back-Translation Still Matters: Leveraging Monolingual Data for Low-Resource Machine Translation with LLMs](https://mlsanity.com/paper/2610.18818/) (2026)

## Abstract (arXiv)

Large language models (LLMs), despite their ability to perform few-shot machine translation (MT), often lag behind dedicated MT systems trained on parallel corpora, which are crucial for high quality machine translation (MT). However, parallel corpora are often scarce or non-existent for low-resource languages. In this paper, we propose CycleDistill, a bootstrapping approach leveraging LLMs and few-shot translation to obtain high-quality MT systems. CycleDistill involves iteratively generating synthetic parallel corpora from monolingual corpora via zero- or few-shot MT, which is then used to fine-tune the model that was used for generating said data for MT. CycleDistill does not need parallel corpora beyond 1 to 4 few-shot examples, and in our experiments focusing on three Indian languages, by relying solely on monolingual corpora, it can achieve high-quality machine translation, improving upon a few-shot baseline model by over 20-30 chrF points on average in the first iteration. We also study the effect of leveraging softmax activations during the distillation process and observe mild improvements in translation quality.