mlsanityData to 7 Oct 2026

Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation

arXiv 2507.22608 ↗First posted 30 Jul 2025cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit strong multilingual abilities, yet the neural mechanisms behind language-specific processing remain unclear. We analyze language-specific neurons in Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse-8B & 32B across 21 typologically diverse languages, identifying neurons that control language behavior. Using the Language Activation Probability Entropy (LAPE) method, we show that these neurons cluster in deeper layers, with non-Latin scripts showing greater specialization. Related languages share overlapping neurons, reflecting internal representations of linguistic proximity. Through language arithmetics, i.e. systematic activation addition and multiplication, we steer models to deactivate unwanted languages and activate desired ones, outperforming simpler replacement approaches. These interventions effectively guide behavior across five multilingual tasks: language forcing, translation, QA, comprehension, and NLI. Manipulation is more successful for high-resource languages, while typological similarity improves effectiveness. We also demonstrate that cross-lingual neuron steering enhances downstream performance and reveal internal "fallback" mechanisms for language selection when neurons are progressively deactivated. Our code is made publicly available at https://github.com/d-gurgurov/Language-Neurons-Manipulation.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

23 of 36 references.

References · 23

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2020Artificial Intelligence, Values and Alignmentcited by 443
  12. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
11 more
  1. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  2. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  3. 2024How do Large Language Models Handle Multilingualism?cited by 148
  4. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  5. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  6. 2025Do Multilingual LLMs Think In English?cited by 75
  7. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  8. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  9. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steeringcited by 12
  10. 2025Language-specific Neurons Do Not Facilitate Cross-Lingual Transfercited by 11
  11. 2025Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languagescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  2. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 11
  3. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  4. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  5. 2026NEAT: Neuron-Based Early Exit for Large Reasoning Modelscited by 3
  6. 2026Language Steering for Multilingual In-Context Learningcited by 3
  7. 2026One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Mergingcited by 2
  8. 2026Multilingual Language Models Encode Script Over Linguistic Structurecited by 2
  9. 2025Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Modelscited by 1
  10. 2026Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Modelscited by 0
  11. 2026The Latin Substrate: How Language Models Represent and Mediate Script Choicecited by 0
  12. 2026Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuningcited by 0
5 more
  1. 2026Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generationcited by 0
  2. 2026Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMscited by 0
  3. 2026Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMscited by 0
  4. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0
  5. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0