mlsanityData to 14 Oct 2026

QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs

arXiv 2412.11763 ↗First posted 16 Dec 2024cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 55 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

19 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey17 Feb
  2. 2023Understanding Social Reasoning in Language Models with Language Models21 Jun

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  7. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  8. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  9. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  10. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  11. 2023What's In My Big Data?cited by 129
  12. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
5 more
  1. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
  2. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
  3. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 45
  4. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  5. 2024A Survey of Useful LLM Evaluationcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  2. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)
  3. 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
  4. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  5. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  6. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  7. 2025Reasoning Capabilities and Invariability of Large Language Models
  8. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains