mlsanityData to 7 Oct 2026

The Larger the Better? Improved LLM Code-Generation via Budget Reallocation

arXiv 2404.00725 ↗First posted 31 Mar 2024cs.SE cs.AI cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens when both models operate under the same budget? (e.g., compute, run-time). To address this question, we analyze code generation LLMs of various sizes and make comparisons such as running a 70B model once vs. generating five outputs from a 13B model. We consider a standard unit-test setup, which can be used to select the correct output from the smaller model. Our findings reveal that the repeated use of smaller models can yield consistent improvements, with gains of up to 15% across five tasks. On the other hand, in scenarios where unit-tests are unavailable, a ranking-based selection of candidates from the smaller model falls short of the performance of a single output from larger ones. Our results highlight the potential of using smaller models instead of larger ones, and the importance of studying approaches for ranking LLM outputs.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

39 of 47 references.

References · 39

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
27 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  10. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  12. 2021Scaling Laws for Transfercited by 265
  13. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  14. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  15. 2019SPoC: Search-based Pseudocode to Codecited by 176
  16. 2023Speculative Decoding with Big Little Decodercited by 163
  17. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  18. 2022Natural Language to Code Translation with Executioncited by 122
  19. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 108
  20. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  21. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  22. 2023Textually Pretrained Speech Language Modelscited by 99
  23. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  24. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  25. 2024When Do We Not Need Larger Vision Models?cited by 65
  26. 2021Pre-trained Language Model based Ranking in Baidu Searchcited by 33
  27. 2023Scaling Laws for Discriminative Speech Recognition Rescoring Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 20; 2026: 8.

41 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2024AI Agents That Mattercited by 199
  4. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  5. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2025S*: Test Time Scaling for Code Generationcited by 95
  8. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 82
  9. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  10. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  11. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  12. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
29 more
  1. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  2. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  3. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  4. 2024PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedbackcited by 22
  5. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  6. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 14
  7. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  8. 2024Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Modelscited by 9
  9. 2026Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimizationcited by 8
  10. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  11. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  12. 2025HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claimscited by 4
  13. 2025Pretraining Scaling Laws for Generative Evaluations of Language Modelscited by 4
  14. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Taskscited by 3
  15. 2025Wisdom and Delusion of LLM Ensembles for Code Generation and Repaircited by 3
  16. 2024Can LLMs plan paths in the real world?cited by 2
  17. 2025Integrating Language Models for Enhanced Network State Monitoring in DRL-Based SFC Provisioningcited by 2
  18. 2025LiLM-RDB-SFC: Lightweight Language Model with Relational Database-Guided DRL for Optimized SFC Provisioningcited by 2
  19. 2026Self-Execution Simulation Improves Coding Modelscited by 2
  20. 2025Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factualitycited by 1
  21. 2025CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfercited by 1
  22. 2025Analysing Environmental Efficiency in AI for X-Ray Diagnosiscited by 1
  23. 2026More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)cited by 1
  24. 2026Computational Arbitrage in AI Model Marketscited by 1
  25. 2025Reasoning Distillation and Structural Alignment for Improved Code Generationcited by 0
  26. 2026Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoningcited by 0
  27. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  28. 2026Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomycited by 0
  29. 2026CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agentscited by 0