# Unveiling Linguistic Regions in Large Language Models

arXiv: https://arxiv.org/abs/2402.14700 · HTML: https://mlsanity.com/paper/2402.14700/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [Skywork: A More Open Bilingual Foundation Model](https://mlsanity.com/paper/2310.19341/) (2023)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)

## Built on by (most cited first, 22 of 22)

- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction](https://mlsanity.com/paper/2505.20279/) (2025)
- [Systematic Outliers in Large Language Models](https://mlsanity.com/paper/2502.06415/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models](https://mlsanity.com/paper/2410.01335/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Knowledge Localization: Mission Not Accomplished? Enter Query Localization!](https://mlsanity.com/paper/2405.14117/) (2024)
- [Are formal and functional linguistic mechanisms dissociated in language models?](https://mlsanity.com/paper/2503.11302/) (2025)
- [KIF: Knowledge Identification and Fusion for Language Model Continual Learning](https://mlsanity.com/paper/2408.05200/) (2024)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Module-Aware Parameter-Efficient Machine Unlearning on Transformers](https://mlsanity.com/paper/2508.17233/) (2025)
- [Component-Level Lesioning of Language Models Reveals Clinically Aligned Aphasia Phenotypes](https://mlsanity.com/paper/2601.19723/) (2026)
- [EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks](https://mlsanity.com/paper/2604.09535/) (2026)
- [The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare](https://mlsanity.com/paper/2411.03287/) (2024)
- [The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model](https://mlsanity.com/paper/2412.07298/) (2024)
- [Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence](https://mlsanity.com/paper/2608.12036/) (2026)
- [Understanding Textual Capability Degradation in Speech LLMs via Parameter Importance Analysis](https://mlsanity.com/paper/2509.23755/) (2025)
- [MEASER: Malware embedding attacks on open-source LLMs](https://mlsanity.com/paper/2510.10486/) (2025)
- [Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment](https://mlsanity.com/paper/2604.10627/) (2026)
- [Do Language Models Encode Knowledge of Linguistic Constraint Violations?](https://mlsanity.com/paper/2605.12055/) (2026)
- [SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution](https://mlsanity.com/paper/2609.03813/) (2026)
- [ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training](https://mlsanity.com/paper/2609.16639/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs' cross-lingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve cross-lingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependence, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct monolingual regions exist for different languages, and disruption to these specific regions substantially reduces the LLMs' proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common phenomenon observed during further pre-training of LLMs. Overall, exploring the LLMs' functional regions provides insights into the foundation of their intelligence.