mlsanityData to 7 Oct 2026

A2utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction

arXiv 2607.02141 ↗First posted 2 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Most LP-from-text benchmarks are static datasets of word problems written and labeled by hand. Once such a dataset is released, its size is fixed, its difficulty is fixed, and every problem can leak into the training data of future LLMs. We present textbfA2utoLPBench, a benchmark for testing LLM-driven agents on linear programming problems written in plain text. We first pick a feasible point and dual, then write down a problem for which that point is optimal and the objective value is known. The answer is known by construction, with no solver call and no human annotator. The evaluation environment bundles a reference solver-critic baseline and a Docker image whose usage instructions are written for an LLM-driven agent to read. With these in place, any agent can run the benchmark and get a calibrated score with one command. Because the benchmark is a generator rather than a fixed dataset, it has properties no fixed dataset can match: an unlimited supply of fresh problems, a difficulty knob set by (n,m), ground-truth answers correct by construction, low LLM-side cost per problem relative to human authoring, repeatable scores across independent batches, and resistance to training-data leakage when fresh post-cutoff seed ranges are used.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 35 of its 41 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

35 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents23 Apr
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation18 Feb
  4. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 31

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2021Convex optimizationcited by 2,479
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
19 more
  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2022PAL: Program-aided Language Modelscited by 635
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  5. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  7. 2023Faithful Chain-of-Thought Reasoningcited by 285
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  11. 2024OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Modelscited by 105
  12. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  13. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  14. 2024OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modelingcited by 66
  15. 2021Inverse Optimization: Theory and Applicationscited by 48
  16. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 46
  17. 2023OptiMUS: Optimization Modeling Using MIP Solvers and large language modelscited by 39
  18. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  19. 2024OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scalecited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  2. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  3. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  4. 2026Benchmark Everything Everywhere All at Once
  5. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  6. 2026Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
  7. 2026FormulaCode: Evaluating Agentic Optimization on Large Codebases
  8. 2025Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generation