# COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation

arXiv: https://arxiv.org/abs/2608.04336 · HTML: https://mlsanity.com/paper/2608.04336/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 30 of 33)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution](https://mlsanity.com/paper/2509.19349/) (2025)
- [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://mlsanity.com/paper/2604.25850/) (2026)
- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [The Impact of Automated Parameter Optimization on Defect Prediction Models](https://mlsanity.com/paper/1801.10270/) (2018)
- [Optimizing Temperature for Language Models with Multi-Sample Inference](https://mlsanity.com/paper/2502.05234/) (2025)
- [Revisit Self-Debugging with Self-Generated Tests for Code Generation](https://mlsanity.com/paper/2501.12793/) (2025)
- [Issue Localization via LLM-Driven Iterative Code Graph Searching](https://mlsanity.com/paper/2503.22424/) (2025)
- [Cost-Aware Contrastive Routing for LLMs](https://mlsanity.com/paper/2508.12491/) (2025)
- [AgentConductor: Topology Evolution for Multi-Agent Competition-Level Code Generation](https://mlsanity.com/paper/2602.17100/) (2026)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective](https://mlsanity.com/paper/2508.01443/) (2025)
- [Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation](https://mlsanity.com/paper/2505.07270/) (2025)
- [Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks](https://mlsanity.com/paper/2511.04355/) (2025)
- [Evolving Excellence: Automated Optimization of LLM-based Agents](https://mlsanity.com/paper/2512.09108/) (2025)

## Abstract (arXiv)

Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one fixed configuration for all tasks: global optimizers search one configuration for all tasks, routers choose only a model, and prompt optimizers keep the model and decoding settings fixed. This leaves their joint, group-specific interactions unclear. We therefore examine how these choices interact and observe that prompts and decoding settings interact, tuning effects vary by model, and the best configuration varies by task difficulty. Guided by these observations, we introduce COMPAS (Code-generation Optimization over Models, Prompts, And Decoding Settings), a difficulty-aware method that learns group-specific quality-cost fronts through low-cost model selection and joint prompt-decoding search, then routes each test task to its matching front online without further search. Under a matched search budget on LiveCodeBench, COMPAS improves pass@1 from 45.9% for the best baseline to 52.8% while reducing cost from 36.57 to 4.92. This also transfers to repository-level code generation on SWE-bench, resolving 76.0% of tasks versus 70.0% for the best baseline. Code and the reproducibility artifact are available at https://github.com/gjz78910/COMPAS.