# OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.09092 · HTML: https://mlsanity.com/paper/2511.09092/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch](https://mlsanity.com/paper/2410.13213/) (2024)
- [OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling](https://mlsanity.com/paper/2407.09887/) (2024)
- [OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling](https://mlsanity.com/paper/2502.11102/) (2025)
- [Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency](https://mlsanity.com/paper/2309.17272/) (2023)
- [OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents](https://mlsanity.com/paper/2504.16918/) (2025)

## Built on by (most cited first, 17 of 17)

- [Strategy-Aware Optimization Modeling with Reasoning LLMs](https://mlsanity.com/paper/2605.02545/) (2026)
- [Enhancing Table Reasoning with Deterministic Table-State Rewards](https://mlsanity.com/paper/2601.22530/) (2026)
- [MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Research](https://mlsanity.com/paper/2602.03318/) (2026)
- [Execution-Verified Reinforcement Learning for Optimization Modeling](https://mlsanity.com/paper/2604.00442/) (2026)
- [OptSkills: Learning Generalizable Optimization Skills from Problem Archetypes via Cluster-Based Distillation](https://mlsanity.com/paper/2605.29829/) (2026)
- [Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problems](https://mlsanity.com/paper/2606.04816/) (2026)
- [AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problems](https://mlsanity.com/paper/2604.16804/) (2026)
- [Automated Reformulation of Robust Optimization via Memory-Augmented Large Language Models](https://mlsanity.com/paper/2605.11813/) (2026)
- [StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modeling](https://mlsanity.com/paper/2606.15197/) (2026)
- [MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources](https://mlsanity.com/paper/2606.25832/) (2026)
- [PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language](https://mlsanity.com/paper/2607.18256/) (2026)
- [SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD](https://mlsanity.com/paper/2607.20145/) (2026)
- [EvoOpt-LLM: Evolving industrial optimization models with large language models](https://mlsanity.com/paper/2602.01082/) (2026)
- [COOPA: A Modular LLM Agent Architecture for Operations Research Problems](https://mlsanity.com/paper/2606.27611/) (2026)
- [Uncertainty-Aware Simulation-Based Inference for Operations Research with Large Language Models](https://mlsanity.com/paper/2608.00019/) (2026)
- [OptiSkill: A Hierarchical and Evolving SkillBank for LLM-Based Optimization Modeling](https://mlsanity.com/paper/2609.22987/) (2026)
- [Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling](https://mlsanity.com/paper/2609.38884/) (2026)

## Abstract (arXiv)

Optimization modeling and solving are fundamental to the application of Operations Research (OR) in real-world decision making, yet the process of translating natural language problem descriptions into formal models and solver code remains highly expertise intensive. While recent advances in large language models (LLMs) have opened new opportunities for automation, the generalization ability and data efficiency of existing LLM-based methods are still limited, asmost require vast amounts of annotated or synthetic data, resulting in high costs and scalability barriers. In this work, we present OR-R1, a data-efficient training framework for automated optimization modeling and solving. OR-R1 first employs supervised fine-tuning (SFT) to help the model acquire the essential reasoning patterns for problem formulation and code generation from limited labeled data. In addition, it improves the capability and consistency through Test-Time Group Relative Policy Optimization (TGRPO). This two-stage design enables OR-R1 to leverage both scarce labeled and abundant unlabeled data for effective learning. Experiments show that OR-R1 achieves state-of-the-art performance with an average solving accuracy of 67.7%, using only 1/10 the synthetic data required by prior methods such as ORLM, exceeding ORLM's solving accuracy by up to 4.2%. Remarkably, OR-R1 outperforms ORLM by over 2.4% with just 100 synthetic samples. Furthermore, TGRPO contributes an additional 3.1%-6.4% improvement in accuracy, significantly narrowing the gap between single-attempt (Pass@1) and multi-attempt (Pass@8) performance from 13% to 7%. Extensive evaluations across diverse real-world benchmarks demonstrate that OR-R1 provides a robust, scalable, and cost-effective solution for automated OR optimization problem modeling and solving, lowering the expertise and data barriers for industrial OR applications.