mlsanityData to 14 Oct 2026

Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation

arXiv 2507.13238 ↗First posted 17 Jul 2025cs.CL cs.AIcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Analogies test a model's ability to infer implicit relationships between concepts, making them a key benchmark for evaluating reasoning capabilities. While large language models (LLMs) are widely evaluated for reasoning in English, their abilities in Indic languages remain understudied, limiting our understanding of whether these models generalize across languages. To address this gap, we introduce a new Hindi Analogy Test Set (HATS), comprising 405 multiple-choice questions sourced from Indian government exams. We benchmark state-of-the-art multilingual LLMs using various prompting strategies and introduce a grounded Chain of Thought approach that leverages cognitive theories of analogical reasoning. This approach improves model performance on Hindi analogy questions. Our experiments show that models perform best with English prompts, irrespective of the prompting strategy. Our test set addresses the lack of a critical resource to evaluate LLM reasoning capabilities in Hindi.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 25 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

13 of 25 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  4. 2018Learning Word Vectors for 157 Languagescited by 280
  5. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  6. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  7. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  8. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  9. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  10. 2022Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTcited by 27
  11. 2024Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuningcited by 14
  12. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patternscited by 13
1 more
  1. 2018A Knowledge Hunting Framework for Common Sense Reasoningcited by 10

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddles
  2. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  3. 2024Inductive Linguistic Reasoning with Large Language Models
  4. 2026IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
  5. 2025Could Thinking Multilingually Empower LLM Reasoning?
  6. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  7. 2026Rethinking the Multilingual Reasoning Gap with Layer Swap
  8. 2026ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation