# Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization

arXiv: https://arxiv.org/abs/2508.18091 · HTML: https://mlsanity.com/paper/2508.18091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 93 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch](https://mlsanity.com/paper/2410.13213/) (2024)
- [OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling](https://mlsanity.com/paper/2407.09887/) (2024)
- [Large Language Models for Supply Chain Optimization](https://mlsanity.com/paper/2307.03875/) (2023)
- [LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages](https://mlsanity.com/paper/2405.13144/) (2024)
- [Chain-of-Retrieval Augmented Generation](https://mlsanity.com/paper/2501.14342/) (2025)
- [Autoformulation of Mathematical Optimization Models Using LLMs](https://mlsanity.com/paper/2411.01679/) (2024)
- [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](https://mlsanity.com/paper/2505.11792/) (2025)
- [OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale](https://mlsanity.com/paper/2407.19633/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming](https://mlsanity.com/paper/2410.12112/) (2024)
- [Towards Foundation Models for Mixed Integer Linear Programming](https://mlsanity.com/paper/2410.08288/) (2024)
- [Multi-Agent Collaboration via Cross-Team Orchestration](https://mlsanity.com/paper/2406.08979/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [Synthesizing mixed-integer linear programming models from natural language descriptions](https://mlsanity.com/paper/2311.15271/) (2023)
- [BPP-Search: Enhancing Tree of Thought Reasoning for Mathematical Modeling Problem Solving](https://mlsanity.com/paper/2411.17404/) (2024)
- [OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM](https://mlsanity.com/paper/2503.10009/) (2025)
- [Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms](https://mlsanity.com/paper/2410.20848/) (2024)
- [Starjob: Dataset for LLM-Driven Job Shop Scheduling](https://mlsanity.com/paper/2503.01877/) (2025)
- [From Large Language Models and Optimization to Decision Optimization CoPilot: A Research Manifesto](https://mlsanity.com/paper/2402.16269/) (2024)
- [LLMs can Schedule](https://mlsanity.com/paper/2408.06993/) (2024)
- [Leveraging Large Language Models for Solving Rare MIP Challenges](https://mlsanity.com/paper/2409.04464/) (2024)
- [ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research](https://mlsanity.com/paper/2506.01326/) (2025)
- [Decision Information Meets Large Language Models: The Future of Explainable Operations Research](https://mlsanity.com/paper/2502.09994/) (2025)
- [City-LEO: Toward Transparent City Management Using LLM with End-to-End Optimization](https://mlsanity.com/paper/2406.10958/) (2024)

## Built on by (most cited first, 1 of 1)

- [SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations](https://mlsanity.com/paper/2511.00685/) (2025)

## Abstract (arXiv)

This paper investigates the capabilities of large language models (LLMs) in formulating and solving decision-making problems using mathematical programming. We first conduct a systematic review and meta-analysis of recent literature to assess how well LLMs understand, structure, and solve optimization problems across domains. The analysis is guided by critical review questions focusing on learning approaches, dataset designs, evaluation metrics, and prompting strategies. Our systematic evidence is complemented by targeted experiments designed to evaluate the performance of state-of-the-art LLMs in automatically generating optimization models for problems in computer networks. Using a newly constructed dataset, we apply three prompting strategies: Act-as-expert, chain-of-thought, and self-consistency, and evaluate the obtained outputs based on optimality gap, token-level F1 score, and compilation accuracy. Results show promising progress in LLMs' ability to parse natural language and represent symbolic formulations, but also reveal key limitations in accuracy, scalability, and interpretability. These empirical gaps motivate several future research directions, including structured datasets, domain-specific fine-tuning, hybrid neuro-symbolic approaches, modular multi-agent architectures, and dynamic retrieval via chain-of-RAGs. This paper contributes a structured roadmap for advancing LLM capabilities in mathematical programming.