# Understanding the role of FFNs in driving multilingual behaviour in LLMs

arXiv: https://arxiv.org/abs/2404.13855 · HTML: https://mlsanity.com/paper/2404.13855/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models](https://mlsanity.com/paper/2210.14328/) (2022)
- [Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks](https://mlsanity.com/paper/2310.15552/) (2023)
- [A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT](https://mlsanity.com/paper/2004.09205/) (2020)

## Built on by (most cited first, 3 of 3)

- [Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis](https://mlsanity.com/paper/2502.11812/) (2025)
- [How Programming Concepts and Neurons Are Shared in Code Language Models](https://mlsanity.com/paper/2506.01074/) (2025)
- [Exploring the Translation Mechanism of Large Language Models](https://mlsanity.com/paper/2502.11806/) (2025)

## Abstract (arXiv)

Multilingualism in Large Language Models (LLMs) is an yet under-explored area. In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of a Large Language Model, examining its architecture, activation patterns, and processing mechanisms across languages. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing. Our findings reveal different patterns of multilinugal processing in the sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the phenomenon of "over-layerization" in certain model configurations, where increasing layer depth without corresponding adjustments to other parameters may degrade model performance. Through comparisons within and across languages, we demonstrate the interplay between model architecture, layer depth, and multilingual processing capabilities of LLMs trained on multiple languages.