mlsanityData to 7 Oct 2026

LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation

arXiv 2403.01131 ↗First posted 2 Mar 2024math.OC cs.AI cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Recent research explores optimization using large language models (LLMs) by either iteratively seeking next-step solutions from LLMs or directly prompting LLMs for an optimizer. However, these approaches exhibit inherent limitations, including low operational efficiency, high sensitivity to prompt design, and a lack of domain-specific knowledge. We introduce LLaMoCo, the first instruction-tuning framework designed to adapt LLMs for solving optimization problems in a code-to-code manner. Specifically, we establish a comprehensive instruction set containing well-described problem prompts and effective optimization codes. We then develop a novel two-phase learning strategy that incorporates a contrastive learning-based warm-up procedure before the instruction-tuning phase to enhance the convergence behavior during model fine-tuning. The experiment results demonstrate that a CodeGen (350M) model fine-tuned by our LLaMoCo achieves superior optimization performance compared to GPT-4 Turbo and the other competitors across both synthetic and realistic problem sets. The fine-tuned model and the usage instructions are available at https://anonymous.4open.science/r/LLaMoCo-722A.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 69 of its 150 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

69 of 150 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2021Convex optimizationcited by 2,479
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  13. 2023LIMA: Less Is More for Alignmentcited by 1,080
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  15. 2023Large Language Models as Optimizerscited by 830
  16. 2023StarCoder: may the source be with you!cited by 790
  17. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  18. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  19. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  20. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  21. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  22. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  23. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  24. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  25. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  26. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  27. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  28. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  29. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  30. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  31. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  32. 2022Evolution through Large Modelscited by 131
  33. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  34. 2023Large Language Models as Evolutionary Optimizerscited by 122
  35. 2018Automated Algorithm Selection: Survey and Perspectivescited by 111
  36. 2023On the Exploitability of Instruction Tuningcited by 110
  37. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  38. 2020Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Searchcited by 87

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 17; 2026: 17.

45 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2024LLMOPT: Learning to Define and Solve General Optimization Problems from Scratchcited by 102
  3. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  4. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  5. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  6. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  7. 2024ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learningcited by 15
  8. 2024Pretrained Optimization Model for Zero-Shot Black Box Optimizationcited by 13
  9. 2025Planning of Heuristics: Strategic Planning on Large Language Models with Monte Carlo Tree Search for Automating Heuristic Optimizationcited by 13
  10. 2025Reinforcement Learning-based Self-adaptive Differential Evolution through Automated Landscape Feature Learningcited by 12
  11. 2025SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Searchcited by 10
  12. 2025MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimizationcited by 9
33 more
  1. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  2. 2024LLASP: Fine-tuning Large Language Models for Answer Set Programmingcited by 8
  3. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  4. 2024Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approachcited by 6
  5. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  6. 2025Instance Generation for Meta-Black-Box Optimization through Latent Space Reverse Engineeringcited by 6
  7. 2025DesignX: Human-Competitive Algorithm Designer for Black-Box Optimizationcited by 5
  8. 2025An Agentic Framework with LLMs for Solving Complex Vehicle Routing Problemscited by 5
  9. 2026Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verificationcited by 5
  10. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  11. 2024Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcementcited by 3
  12. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  13. 2026TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Designcited by 3
  14. 2026ORPilot: A Production-Oriented Agentic LLM-for-OR Tool for Optimization Modelingcited by 2
  15. 2026OptSkills: Learning Generalizable Optimization Skills from Problem Archetypes via Cluster-Based Distillationcited by 2
  16. 2025OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modelingcited by 1
  17. 2025A Stochastic Differential Equation Framework for Multi-Objective LLM Interactions: Dynamical Systems Analysis with Code Generation Applicationscited by 1
  18. 2025AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solvingcited by 1
  19. 2025Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Designcited by 1
  20. 2026Detect and Act: Automated Dynamic Optimizer through Meta-Black-Box Optimizationcited by 1
  21. 2026Beyond the Node: Clade-level Selection for Efficient MCTS in Automatic Heuristic Designcited by 1
  22. 2026MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resourcescited by 1
  23. 2025Language-Guided Tuning: Configuration Optimization for Automated ML Researchcited by 0
  24. 2026READY: Reward Discovery for Meta-Black-Box Optimizationcited by 0
  25. 2026EvoOpt-LLM: Evolving industrial optimization models with large language modelscited by 0
  26. 2026NED-Tree: Bridging the Semantic Gap with Nonlinear Element Decomposition Tree for LLM Nonlinear Optimization Modelingcited by 0
  27. 2026BLAST: Benchmarking LLMs with ASP-based Structured Testingcited by 0
  28. 2026Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontierscited by 0
  29. 2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generationcited by 0
  30. 2026RAISE: LLM-based Automated Heuristic Design with Robust Adversary Instance Searchcited by 0
  31. 2026Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discoverycited by 0
  32. 2026Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimizationcited by 0
  33. 2026Learning to Optimize through Solver-Grounded Self-Playcited by 0