# DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problems

arXiv: https://arxiv.org/abs/2506.06052 · HTML: https://mlsanity.com/paper/2506.06052/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [S*: Test Time Scaling for Code Generation](https://mlsanity.com/paper/2502.14382/) (2025)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [Complementary Explanations for Effective In-Context Learning](https://mlsanity.com/paper/2211.13892/) (2022)
- [Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions](https://mlsanity.com/paper/2209.15565/) (2022)
- [Leveraging Large Language Models to Generate Answer Set Programs](https://mlsanity.com/paper/2307.07699/) (2023)
- [CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization](https://mlsanity.com/paper/2504.04310/) (2025)
- [Holy Grail 2.0: From Natural Language to Constraint Models](https://mlsanity.com/paper/2308.01589/) (2023)
- [PyCSP3: Modeling Combinatorial Constrained Problems in Python](https://mlsanity.com/paper/2009.00326/) (2020)
- [Synthesis of Mathematical programs from Natural Language Specifications](https://mlsanity.com/paper/2304.03287/) (2023)
- [Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc](https://mlsanity.com/paper/2503.10642/) (2025)

## Built on by (most cited first, 4 of 4)

- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Gala: Global LLM Agents for Text-to-Model Translation](https://mlsanity.com/paper/2509.08970/) (2025)
- [Can Large Language Models Reason and Optimize Under Constraints?](https://mlsanity.com/paper/2603.23004/) (2026)
- [Auto-Formalizing Neuro-Symbolic Predictors](https://mlsanity.com/paper/2610.01519/) (2026)

## Abstract (arXiv)

Discrete Combinatorial Problems (DCPs) are prevalent in industrial decision-making and optimisation. However, while constraint solving technologies for DCPs have advanced significantly, the core process of formalising them, namely constraint modelling, requires significant expertise and remains a bottleneck for wider adoption. Aiming to alleviate this bottleneck, recent studies have explored using Large Language Models (LLMs) to transform combinatorial problem descriptions into executable constraint models. However, the existing evaluation datasets for discrete constraint modelling are often limited to small, homogeneous, or domain-specific problems, which do not capture the diversity of real-world scenarios. This work addresses this gap by introducing DCP-Bench-Open, a novel benchmark that includes a diverse set of well-known discrete combinatorial problems sourced from the Constraint Programming (CP) and Operations Research (OR) communities, structured explicitly for evaluating LLM-driven constraint modelling. With this dataset, and given the variety of modelling frameworks, we compare and evaluate the modelling capabilities of LLMs for three distinct constraint modelling systems, which vary in abstraction level and underlying syntax. Notably, the results show higher performance when modelling with a high-level Python-based framework. Additionally, we systematically evaluate the use of prompt-based and inference-time compute methods across different LLMs, which further increase accuracy, reaching up to 91% on this highly challenging benchmark. DCP-Bench-Open is publicly available.