mlsanityData to 14 Oct 2026

The Emergence of Abstract Thought in Large Language Models Beyond Any Language

arXiv 2506.09890 ↗First posted 11 Jun 2025cs.CL cs.AIcited by 21

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

As large language models (LLMs) continue to advance, their capacity to function effectively across a diverse range of languages has shown marked improvement. Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts. This has led to the widespread assumption that LLMs may "think" in English. However, more recent results showing strong multilingual performance, even surpassing English performance on specific tasks in other languages, challenge this view. In this work, we find that LLMs progressively develop a core language-agnostic parameter space-a remarkably small subset of parameters whose deactivation results in significant performance degradation across all languages. This compact yet critical set of parameters underlies the model's ability to generalize beyond individual languages, supporting the emergence of abstract thought that is not tied to any specific linguistic system. Specifically, we identify language-related neurons-those are consistently activated during the processing of particular languages, and categorize them as either shared (active across multiple languages) or exclusive (specific to one). As LLMs undergo continued development over time, we observe a marked increase in both the proportion and functional importance of shared neurons, while exclusive neurons progressively diminish in influence. These shared neurons constitute the backbone of the core language-agnostic parameter space, supporting the emergence of abstract thought. Motivated by these insights, we propose neuron-specific training strategies tailored to LLMs' language-agnostic levels at different development stages. Experiments across diverse LLM families support our approach.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 50 of its 67 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

50 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners21 May
  2. 2025Do Multilingual LLMs Think In English?21 Feb
  3. 2024Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts17 Sep
  4. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?20 Aug
  5. 2024AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought20 Jun
  6. 2023Do Multilingual Language Models Think Better in English?2 Aug
  7. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting11 May
  8. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  12. 2019How multilingual is Multilingual BERT?cited by 645
30 more
  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  2. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  3. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  4. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  5. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  7. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  8. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  9. 2024How do Large Language Models Handle Multilingualism?cited by 150
  10. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  11. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  12. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  13. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  14. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  15. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 66
  16. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  17. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  18. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  19. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  20. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 33
  21. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 33
  22. 2024Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Modelscited by 31
  23. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  24. 2021Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Trainingcited by 28
  25. 2024Unified Parameter-Efficient Unlearning for LLMscited by 25
  26. 2024Teaching Llama a New Language Through Cross-Lingual Knowledge Transfercited by 23
  27. 2024How Do Multilingual Language Models Remember Facts?cited by 20
  28. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  29. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16
  30. 2024Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasetscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 17.

21 papers.

Sort:

By area:

  1. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  2. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 21
  3. 2025Parallel Test-Time Scaling for Latent Reasoning Modelscited by 14
  4. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 10
  5. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  6. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 4
  7. 2025Calibrated Multimodal Representation Learning with Missing Modalitiescited by 3
  8. 2026Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfercited by 3
  9. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  10. 2026Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Promptscited by 2
  11. 2026Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Modelscited by 1
  12. 2026From Circuit Evidence to Mechanistic Theory: An Inductive Logic Approachcited by 1
9 more
  1. 2026Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearningcited by 1
  2. 2026Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Headscited by 0
  3. 2026Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Modelscited by 0
  4. 2026Multilingual Fine-Tuning via Localized Gradient Conflict Resolutioncited by 0
  5. 2026One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMscited by 0
  6. 2026SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMscited by 0
  7. 2026Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathwayscited by 0
  8. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognitioncited by 0
  9. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models
  2. 2023Dissociating language and thought in large language models
  3. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units
  4. 2026The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?
  5. 2026Understanding Large Language Models
  6. 2024Probing the Emergence of Cross-lingual Alignment during LLM Training
  7. 2025Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
  8. 2025Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons