# Counter-Interference Adapter for Multilingual Machine Translation

arXiv: https://arxiv.org/abs/2104.08154 · HTML: https://mlsanity.com/paper/2104.08154/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [Generalized Data Augmentation for Low-Resource Translation](https://mlsanity.com/paper/1906.03785/) (2019)
- [Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2005.04816/) (2020)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment](https://mlsanity.com/paper/2010.03017/) (2020)

## Built on by (most cited first, 30 of 40)

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis](https://mlsanity.com/paper/2403.01439/) (2024)
- [Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies](https://mlsanity.com/paper/2410.19878/) (2024)
- [Continual Prompt Tuning for Dialog State Tracking](https://mlsanity.com/paper/2203.06654/) (2022)
- [OWL: A Large Language Model for IT Operations](https://mlsanity.com/paper/2309.09298/) (2023)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [Exploring Efficient-tuning Methods in Self-supervised Speech Models](https://mlsanity.com/paper/2210.06175/) (2022)
- [Learning Kernel-Smoothed Machine Translation with Retrieved Examples](https://mlsanity.com/paper/2109.09991/) (2021)
- [Causes and Cures for Interference in Multilingual Translation](https://mlsanity.com/paper/2212.07530/) (2022)
- [AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping](https://mlsanity.com/paper/2505.18738/) (2025)
- [Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2212.10551/) (2022)
- [Spectral Adapter: Fine-Tuning in Spectral Space](https://mlsanity.com/paper/2405.13952/) (2024)
- [Multilingual Machine Translation with Hyper-Adapters](https://mlsanity.com/paper/2205.10835/) (2022)
- [Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision](https://mlsanity.com/paper/2406.03051/) (2024)
- [Structured Pruning Adapters](https://mlsanity.com/paper/2211.10155/) (2022)
- [Learning Attentional Mixture of LoRAs for Language Model Continual Learning](https://mlsanity.com/paper/2409.19611/) (2024)
- [Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning](https://mlsanity.com/paper/2301.10915/) (2023)
- [On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules](https://mlsanity.com/paper/2204.08653/) (2022)
- [Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony](https://mlsanity.com/paper/2408.09397/) (2024)
- [An Efficient Adaptive Compression Method for Human Perception and Machine Vision Tasks](https://mlsanity.com/paper/2501.04329/) (2025)
- [Memory-Efficient Fine-Tuning of Transformers via Token Selection](https://mlsanity.com/paper/2501.18824/) (2025)
- [On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation](https://mlsanity.com/paper/2505.22444/) (2025)
- [Towards a Deep Understanding of Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/2310.20456/) (2023)
- [PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2312.13066/) (2023)
- [Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images](https://mlsanity.com/paper/2406.02385/) (2024)
- [Unifying the Convergences in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2205.01620/) (2022)
- [Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models](https://mlsanity.com/paper/2408.14470/) (2024)

## Abstract (arXiv)

Developing a unified multilingual model has long been a pursuit for machine translation. However, existing approaches suffer from performance degradation -- a single multilingual model is inferior to separately trained bilingual ones on rich-resource languages. We conjecture that such a phenomenon is due to interference caused by joint training with multiple languages. To accommodate the issue, we propose CIAT, an adapted Transformer model with a small parameter overhead for multilingual machine translation. We evaluate CIAT on multiple benchmark datasets, including IWSLT, OPUS-100, and WMT. Experiments show that CIAT consistently outperforms strong multilingual baselines on 64 of total 66 language directions, 42 of which see above 0.5 BLEU improvement. Our code is available at urlhttps://github.com/Yaoming95/CIAT~.