mlsanityData to 7 Oct 2026

CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization

arXiv 2504.04310 ↗First posted 6 Apr 2025cs.CL cs.AIcited by 28

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Although LLM-based agents have attracted significant attention in domains such as software engineering and machine learning research, their role in advancing combinatorial optimization (CO) remains relatively underexplored. This gap underscores the need for a deeper understanding of their potential in tackling structured, constraint-intensive problems -- a pursuit currently limited by the absence of comprehensive benchmarks for systematic investigation. To address this, we introduce CO-Bench, a benchmark suite featuring 36 real-world CO problems drawn from a broad range of domains and complexity levels. CO-Bench includes structured problem formulations and curated data to support rigorous investigation of LLM agents. We evaluate multiple agentic frameworks against established human-designed algorithms, revealing the strengths and limitations of existing LLM agents and identifying promising directions for future research. CO-Bench is publicly available at https://github.com/sunnweiwei/CO-Bench.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 27 of its 82 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

27 of 82 references.

References · 27

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  9. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 612
  10. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  11. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  12. 2019Exact Combinatorial Optimization with Graph Convolutional Neural Networkscited by 313
15 more
  1. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  2. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  3. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  4. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 216
  5. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  6. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  7. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 133
  8. 2022DIMES: A Differentiable Meta Solver for Combinatorial Optimization Problemscited by 127
  9. 2024OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Modelscited by 105
  10. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 100
  11. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  12. 2024OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modelingcited by 66
  13. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 48
  14. 2022What's Wrong with Deep Learning in Tree Search for Combinatorial Optimizationcited by 36
  15. 2018Packing unequal rectangles and squares in a fixed size circular container using formulation space searchcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 12.

28 papers.

Sort:

By area:

  1. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  2. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  3. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  4. 2025AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Librarycited by 12
  5. 2025FrontierCO: Real-World and Large-Scale Evaluation of Machine Learning Solvers for Combinatorial Optimizationcited by 10
  6. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  7. 2026Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimizationcited by 9
  8. 2025Generalizable Heuristic Generation Through LLMs with Meta-Optimizationcited by 7
  9. 2025VRPAgent: LLM-Driven Discovery of Heuristic Operators for Vehicle Routing Problemscited by 6
  10. 2025CoMind: Towards Community-Driven Agents for Machine Learning Engineeringcited by 5
  11. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  12. 2026FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimizationcited by 4
16 more
  1. 2025Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problemscited by 3
  2. 2025DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problemscited by 3
  3. 2025STRCMP: Integrating Graph Structural Priors with Language Models for Combinatorial Optimizationcited by 3
  4. 2026Metan: Recursive Self-Improvement through Emergent Depthcited by 3
  5. 2026ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issuescited by 2
  6. 2025Attacks and Defenses Against LLM Fingerprintingcited by 1
  7. 2025MCCE: A Framework for Multi-LLM Collaborative Search in Discrete Spaces with Similarity-Filtered Preference Learningcited by 1
  8. 2026Memory-Guided Tree Search with Cross-Branch Knowledge Transfer for LLM Solver Synthesiscited by 1
  9. 2026SpecAHD: Localize to Specialize for Automated Heuristic Design in Large-Scale Routing Problemscited by 1
  10. 2026ATLAS: Scaffold-Free Algorithm Synthesis by LLMs via Embedding-Guided Quality-Diversity Searchcited by 1
  11. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  12. 2025Reasoning with Preference Constraints: A Benchmark for Language Models in Many-to-One Matching Marketscited by 0
  13. 2026LLM Agents for Combinatorial Efficient Frontiers: Investment Portfolio Optimizationcited by 0
  14. 2026ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?cited by 0
  15. 2026RAISE: LLM-based Automated Heuristic Design with Robust Adversary Instance Searchcited by 0
  16. 2026When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization
  2. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  3. 2026DRAGON: LLM-Driven Decomposition and Reconstruction Agents for Large-Scale Combinatorial Optimization
  4. 2026Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
  5. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  6. 2025ACCORD: Autoregressive Constraint-satisfying Generation for COmbinatorial Optimization with Routing and Dynamic attention
  7. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  8. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebases