# Evaluating LLM Reasoning in the Operations Research Domain with ORQA

arXiv: https://arxiv.org/abs/2412.17874 · HTML: https://mlsanity.com/paper/2412.17874/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [GPT-4 Can't Reason](https://mlsanity.com/paper/2308.03762/) (2023)
- [LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond](https://mlsanity.com/paper/2305.14540/) (2023)
- [EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries](https://mlsanity.com/paper/2402.16040/) (2024)
- [A Study on Large Language Models' Limitations in Multiple-Choice Question Answering](https://mlsanity.com/paper/2401.07955/) (2024)
- [Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing](https://mlsanity.com/paper/2305.03195/) (2023)
- [STREET: A Multi-Task Structured Reasoning and Explanation Benchmark](https://mlsanity.com/paper/2302.06729/) (2023)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)

## Built on by (most cited first, 12 of 12)

- [EvoCut: Strengthening Integer Programs via Evolution-Guided Language Models](https://mlsanity.com/paper/2508.11850/) (2025)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Bilevel Optimization of Agent Skills via Monte Carlo Tree Search](https://mlsanity.com/paper/2604.15709/) (2026)
- [Integrating Large Language Models with Network Optimization for Interactive and Explainable Supply Chain Planning: A Real-World Case Study](https://mlsanity.com/paper/2508.21622/) (2025)
- [OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents](https://mlsanity.com/paper/2605.28158/) (2026)
- [Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems](https://mlsanity.com/paper/2510.10644/) (2025)
- [AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems](https://mlsanity.com/paper/2604.16804/) (2026)
- [Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization](https://mlsanity.com/paper/2605.21751/) (2026)
- [OR-Toolformer: Modeling and Solving Operations Research Problems with Tool Augmented Large Language Models](https://mlsanity.com/paper/2510.01253/) (2025)
- [LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocation](https://mlsanity.com/paper/2512.00039/) (2025)
- [LLM Agents for Combinatorial Efficient Frontiers: Investment Portfolio Optimization](https://mlsanity.com/paper/2601.00770/) (2026)
- [Adapting to Evolving Requirements: Agentic AI for Retail Supply Chain Operations](https://mlsanity.com/paper/2609.03860/) (2026)

## Abstract (arXiv)

In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available.