# Large Language Models Are Cross-Lingual Knowledge-Free Reasoners

arXiv: https://arxiv.org/abs/2406.16655 · HTML: https://mlsanity.com/paper/2406.16655/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Do language models plan ahead for future tokens?](https://mlsanity.com/paper/2404.00859/) (2024)
- [Question Translation Training for Better Multilingual Reasoning](https://mlsanity.com/paper/2401.07817/) (2024)
- [Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability](https://mlsanity.com/paper/2306.06688/) (2023)
- [Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly](https://mlsanity.com/paper/2404.04659/) (2024)
- [Crosslingual Generalization through Multitask Finetuning](https://mlsanity.com/paper/2211.01786/) (2022)
- [MTG: A Benchmark Suite for Multilingual Text Generation](https://mlsanity.com/paper/2108.07140/) (2021)
- [An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs](https://mlsanity.com/paper/2406.12288/) (2024)
- [LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations](https://mlsanity.com/paper/2401.12576/) (2024)
- [SambaLingo: Teaching Large Language Models New Languages](https://mlsanity.com/paper/2404.05829/) (2024)
- [What Is Missing in Multilingual Visual Reasoning and How to Fix It](https://mlsanity.com/paper/2403.01404/) (2024)

## Built on by (most cited first, 16 of 16)

- [Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning](https://mlsanity.com/paper/2412.13540/) (2024)
- [MPO: Multilingual Safety Alignment via Reward Gap Optimization](https://mlsanity.com/paper/2505.16869/) (2025)
- [How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective](https://mlsanity.com/paper/2505.21505/) (2025)
- [Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric Phenomenon](https://mlsanity.com/paper/2506.01675/) (2025)
- [MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation](https://mlsanity.com/paper/2510.08608/) (2025)
- [Cross-Lingual Activation Steering for Multilingual Language Models](https://mlsanity.com/paper/2601.16390/) (2026)
- [Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From](https://mlsanity.com/paper/2504.10906/) (2025)
- [Multilingual Knowledge Graph Completion via Efficient Multilingual Knowledge Sharing](https://mlsanity.com/paper/2510.07736/) (2025)
- [Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models](https://mlsanity.com/paper/2605.09496/) (2026)
- [On the Robustness of Knowledge Editing for Detoxification](https://mlsanity.com/paper/2602.10504/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer](https://mlsanity.com/paper/2606.21954/) (2026)
- [Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection](https://mlsanity.com/paper/2608.00207/) (2026)
- [Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders](https://mlsanity.com/paper/2608.23809/) (2026)
- [Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict](https://mlsanity.com/paper/2608.28645/) (2026)
- [Cross-Lingual Alignment for Decoder-Only Models using MoE Routers](https://mlsanity.com/paper/2610.01921/) (2026)

## Abstract (arXiv)

Large Language Models have demonstrated impressive reasoning capabilities across multiple languages. However, the relationship between capabilities in different languages is less explored. In this work, we decompose the process of reasoning tasks into two separated components: knowledge retrieval and knowledge-free reasoning, and analyze the relationship between cross-lingual transferability and these two components. With adapted commonsense reasoning datasets and constructed knowledge-free reasoning datasets, we show that the knowledge-free reasoning capability can be nearly perfectly transferred across various source-target language directions despite the secondary impact of resource in some specific target languages, while cross-lingual knowledge retrieval significantly hinders the transfer. Moreover, by analyzing the hidden states and feed-forward network neuron activation during the reasoning, we show that higher similarity of hidden representations and larger overlap of activated neurons could explain the better cross-lingual transferability of knowledge-free reasoning than knowledge retrieval. Thus, we hypothesize that knowledge-free reasoning shares similar neurons in different languages for reasoning, while knowledge is stored separately in different languages. Our code and data is available at: https://github.com/NJUNLP/Knowledge-Free-Reasoning.