# SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages

arXiv: https://arxiv.org/abs/2210.11621 · HTML: https://mlsanity.com/paper/2210.11621/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Compression of Generative Pre-trained Language Models via Quantization](https://mlsanity.com/paper/2203.10705/) (2022)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers](https://mlsanity.com/paper/2010.03034/) (2020)
- [Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages](https://mlsanity.com/paper/2009.11201/) (2020)
- [Learning Light-Weight Translation Models from Deep Transformer](https://mlsanity.com/paper/2012.13866/) (2020)
- [Attention-Guided Answer Distillation for Machine Reading Comprehension](https://mlsanity.com/paper/1808.07644/) (2018)
- [Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data](https://mlsanity.com/paper/2105.15071/) (2021)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [What Do Compressed Multilingual Machine Translation Models Forget?](https://mlsanity.com/paper/2205.10828/) (2022)

## Built on by (most cited first, 16 of 16)

- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model](https://mlsanity.com/paper/2212.09811/) (2022)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts](https://mlsanity.com/paper/2311.01070/) (2023)
- [Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation](https://mlsanity.com/paper/2305.14016/) (2023)
- [Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin](https://mlsanity.com/paper/2307.00382/) (2023)
- [HistNERo: Historical Named Entity Recognition for the Romanian Language](https://mlsanity.com/paper/2405.00155/) (2024)
- [Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages](https://mlsanity.com/paper/2302.03528/) (2023)
- [Efficient Compression of Multitask Multilingual Speech Models](https://mlsanity.com/paper/2405.00966/) (2024)
- [CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval](https://mlsanity.com/paper/2412.13071/) (2024)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English](https://mlsanity.com/paper/2302.09611/) (2023)
- [Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Research](https://mlsanity.com/paper/2601.09716/) (2025)
- [Comparative Evaluation of Machine Translation Systems on Images with Text](https://mlsanity.com/paper/2605.29476/) (2026)

## Abstract (arXiv)

In recent years, multilingual machine translation models have achieved promising performance on low-resource language pairs by sharing information between similar languages, thus enabling zero-shot translation. To overcome the "curse of multilinguality", these models often opt for scaling up the number of parameters, which makes their use in resource-constrained environments challenging. We introduce SMaLL-100, a distilled version of the M2M-100 (12B) model, a massively multilingual machine translation model covering 100 languages. We train SMaLL-100 with uniform sampling across all language pairs and therefore focus on preserving the performance of low-resource languages. We evaluate SMaLL-100 on different low-resource benchmarks: FLORES-101, Tatoeba, and TICO-19 and demonstrate that it outperforms previous massively multilingual models of comparable sizes (200-600M) while improving inference latency and memory usage. Additionally, our model achieves comparable results to M2M-100 (1.2B), while being 3.6x smaller and 4.3x faster at inference. Code and pre-trained models: https://github.com/alirezamshi/small100