mlsanityData to 7 Oct 2026

Evaluating LLM Reasoning in the Operations Research Domain with ORQA

arXiv 2412.17874 ↗First posted 22 Dec 2024cs.CL cs.AIcited by 12

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 54 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

23 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Mistral 7Bcited by 3,065
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  12. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
10 more
  1. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  2. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  3. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 108
  4. 2023GPT-4 Can't Reasoncited by 39
  5. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 37
  6. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  7. 2024A Study on Large Language Models' Limitations in Multiple-Choice Question Answeringcited by 22
  8. 2023Gpt-4: A Review on Advancements and Opportunities in Natural Language Processingcited by 15
  9. 2023STREET: A Multi-Task Structured Reasoning and Explanation Benchmarkcited by 13
  10. 2024Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Processcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025EvoCut: Strengthening Integer Programs via Evolution-Guided Language Modelscited by 8
  2. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  3. 2025Integrating Large Language Models with Network Optimization for Interactive and Explainable Supply Chain Planning: A Real-World Case Studycited by 3
  4. 2026Bilevel Optimization of Agent Skills via Monte Carlo Tree Searchcited by 2
  5. 2026OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agentscited by 2
  6. 2025Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systemscited by 1
  7. 2026AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problemscited by 1
  8. 2025OR-Toolformer: Modeling and Solving Operations Research Problems with Tool Augmented Large Language Modelscited by 0
  9. 2025LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocationcited by 0
  10. 2026LLM Agents for Combinatorial Efficient Frontiers: Investment Portfolio Optimizationcited by 0
  11. 2026Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimizationcited by 0
  12. 2026Adapting to Evolving Requirements: Agentic AI for Retail Supply Chain Operationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM
  2. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
  3. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch
  4. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  5. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  6. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  7. 2023Competition-Level Problems are Effective LLM Evaluators
  8. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions