# Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation

arXiv: https://arxiv.org/abs/2404.11201 · HTML: https://mlsanity.com/paper/2404.11201/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 71 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Do Current Multi-Task Optimization Methods in Deep Learning Even Help?](https://mlsanity.com/paper/2209.11379/) (2022)
- [Parameter-Efficient Fine-Tuning without Introducing New Latency](https://mlsanity.com/paper/2305.16742/) (2023)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)

## Built on by (most cited first, 19 of 19)

- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Style-Specific Neurons for Steering LLMs in Text Style Transfer](https://mlsanity.com/paper/2410.00593/) (2024)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models](https://mlsanity.com/paper/2505.16538/) (2025)
- [How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective](https://mlsanity.com/paper/2505.21505/) (2025)
- [Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?](https://mlsanity.com/paper/2507.12838/) (2025)
- [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025)
- [Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection](https://mlsanity.com/paper/2605.23036/) (2026)
- [IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translation](https://mlsanity.com/paper/2408.11512/) (2024)
- [Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning](https://mlsanity.com/paper/2502.11364/) (2025)
- [Neuron Empirical Gradient: Discovering and Quantifying Neurons Global Linear Controllability](https://mlsanity.com/paper/2412.18053/) (2024)
- [Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages](https://mlsanity.com/paper/2508.17078/) (2025)
- [Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation](https://mlsanity.com/paper/2409.05224/) (2024)
- [MoMa: A Modular Deep Learning Framework for Material Property Prediction](https://mlsanity.com/paper/2502.15483/) (2025)
- [LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation](https://mlsanity.com/paper/2507.18940/) (2025)
- [Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Models](https://mlsanity.com/paper/2601.17387/) (2026)
- [Consensus-Aligned Neuron Efficient Fine-Tuning Large Language Models for Multi-Domain Machine Translation](https://mlsanity.com/paper/2602.05694/) (2026)
- [Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs](https://mlsanity.com/paper/2609.01356/) (2026)

## Abstract (arXiv)

Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. Language-specific modeling methods show promise in reducing interference. However, they often rely on heuristics to distribute capacity and struggle to foster cross-lingual transfer via isolated modules. In this paper, we explore intrinsic task modularity within multilingual networks and leverage these observations to circumvent interference under multilingual translation. We show that neurons in the feed-forward layers tend to be activated in a language-specific manner. Meanwhile, these specialized neurons exhibit structural overlaps that reflect language proximity, which progress across layers. Based on these findings, we propose Neuron Specialization, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks. Extensive experiments show that our approach achieves consistent performance gains over strong baselines with additional analyses demonstrating reduced interference and increased knowledge transfer.