mlsanityData to 7 Oct 2026

CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

arXiv 2512.21877 ↗First posted 26 Dec 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard web searches. Although LLMs have advanced significantly in Text-to-SQL tasks, their capability to handle domain-specific nuances and multilingual requirements in sports analytics remains under-explored. We present CricBench, a benchmark suite evaluating the intrinsic SQL generation abilities of LLMs on cricket data across four formats: Test, ODI, T20I, and IPL. We curate a Gold-Standard dataset of 2,654 evaluation instances across four languages (English, Hindi, Punjabi, and Telugu). We evaluate seven models, GPT-5 Mini, Claude Sonnet 4, DeepSeek R1 and V3, Qwen 235B, Llama 3.1, and Gemma 2, using schema-only prompting. No single model dominates across all formats: GPT-5 Mini leads on Test cricket (12.4% DMA), Qwen 235B leads on IPL (28.7%) and T20I (17.5%), and all models score 0% on hard ODI queries. All models show a stark disconnect between syntactic validity (>98% execution accuracy) and semantic correctness (<29% DMA), with a domain gap of 37-55 percentage points versus BIRD. To our knowledge, CricBench is the first Text-to-SQL benchmark for cricket analytics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 26 Dec 2025.

Builds on

17 of 31 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  12. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
5 more
  1. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  2. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  3. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  4. 2019X-SQL: reinforce schema representation with contextcited by 22
  5. 2024On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQLcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
  2. 2024BADGE: BADminton report Generation and Evaluation with LLM
  3. 2024SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs
  4. 2025SPORTSQL: An Interactive System for Real-Time Sports Reasoning and Visualization
  5. 2026Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
  6. 2025IndicRAGSuite: Large-Scale Datasets and a Benchmark for Indian Language RAG Systems
  7. 2025Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation
  8. 2025Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs