# StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modeling

arXiv: https://arxiv.org/abs/2606.15197 · HTML: https://mlsanity.com/paper/2606.15197/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Scalable Best-of-N Selection for Large Language Models via Self-Certainty](https://mlsanity.com/paper/2502.18581/) (2025)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch](https://mlsanity.com/paper/2410.13213/) (2024)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [ThetaEvolve: Test-time Learning on Open Problems](https://mlsanity.com/paper/2511.23473/) (2025)
- [Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions](https://mlsanity.com/paper/2209.15565/) (2022)
- [Test-Time Learning for Large Language Models](https://mlsanity.com/paper/2505.20633/) (2025)
- [OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling](https://mlsanity.com/paper/2502.11102/) (2025)
- [Autoformulation of Mathematical Optimization Models Using LLMs](https://mlsanity.com/paper/2411.01679/) (2024)
- [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](https://mlsanity.com/paper/2505.11792/) (2025)
- [OptiTree: Hierarchical Thoughts Generation with Tree Search for LLM Optimization Modeling](https://mlsanity.com/paper/2510.22192/) (2025)
- [LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems](https://mlsanity.com/paper/2403.01342/) (2024)
- [OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learning](https://mlsanity.com/paper/2511.09092/) (2025)
- [BPP-Search: Enhancing Tree of Thought Reasoning for Mathematical Modeling Problem Solving](https://mlsanity.com/paper/2411.17404/) (2024)
- [ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research](https://mlsanity.com/paper/2506.01326/) (2025)
- [SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search](https://mlsanity.com/paper/2510.16916/) (2025)
- [Tool Verification for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2603.02203/) (2026)
- [SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling](https://mlsanity.com/paper/2510.05115/) (2025)
- [ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization](https://mlsanity.com/paper/2602.15983/) (2026)

## Built on by (most cited first, 1 of 1)

- [MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation](https://mlsanity.com/paper/2606.28876/) (2026)

## Abstract (arXiv)

Optimization modeling is inherently hierarchical, requiring a precise sequence of symbolic commitments. Traditional learning-based automated optimization modeling methods improve modeling policies through large-scale annotated or curated training data, but are costly to adapt to new problem distributions. Meanwhile, one-shot generation remains brittle in hierarchical modeling, where early symbolic errors can propagate into invalid formulations. Test-time scaling offers a promising alternative by enabling structural exploration with additional instance-level computation; however, existing search-based methods typically rely on a fixed policy, causing repeated rollouts to inherit similar modeling biases and providing limited credit assignment for intermediate decisions. To address these limitations, we propose StarOR, a synergistic search-and-adaptation framework that couples MCTS with Test-Time Reinforcement Learning for optimization modeling. StarOR decomposes the modeling process into four stages and updates a transient LoRA adapter via GRPO at each non-terminal node. By using MCTS-generated siblings as local comparison sets, StarOR transforms search-time exploration into instance-specific policy refinement. Moreover, an unsupervised multi-faceted reward system provides fine-grained feedback for intermediate formulation decisions without ground-truth labels. Across five optimization benchmarks, StarOR attains a 65.0% average accuracy with a 4B backbone, matching the highest average and outperforming the evaluated same-backbone test-time baselines. Code is available at hrefhttps://github.com/Liwow/StarORStarOR.