Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders
On the map: Mechanistic interpretability → SAE features and their applications
Abstract (arXiv)
Multilingual large language models (LLMs) exhibit strong cross-linguistic generalization, yet medium to low resource languages underperform on common benchmarks such as ARC-Challenge, MMLU, and HellaSwag. We analyze activation patterns in Gemma-2-2B across all 26 residual layers and 10 languages: Chinese (zh), Russian (ru), Spanish (es), Italian (it), medium to low resource languages including Indonesian (id), Catalan (ca), Marathi (mr), Malayalam (ml), and Hindi (hi), with English (en) as the reference. Using Sparse Autoencoders (SAEs), we reveal systematic disparities in activation patterns. Medium to low resource languages receive up to 26.27 percent lower activations in early layers, with a persistent gap of 19.89 percent in deeper layers. To address this, we apply activation-aware fine-tuning via Low-Rank Adaptation (LoRA), leading to substantial activation gains, such as 87.69 percent for Malayalam and 86.32 percent for Hindi, while maintaining English retention at approximately 91 percent. After fine-tuning, benchmark results show modest but consistent improvements, highlighting activation alignment as a key factor in enhancing multilingual LLM performance.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 15 of its 27 references that are in the map. First posted on arXiv 25 Jul 2025.
Builds on
15 of 27 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2019How multilingual is Multilingual BERT?cited by 645
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
- 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
- 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
- 2022Democratizing Neural Machine Translation with OPUS-MTcited by 58
- 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
- 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 23
- 2020Identifying Necessary Elements for BERT's Multilingualitycited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
- 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders
- 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
- 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
- 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
- 2024Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
- 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks
- 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity