# B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests arXiv: https://arxiv.org/abs/2409.08692 ยท HTML: https://mlsanity.com/paper/2409.08692/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 48 references that are in the map. - First posted on arXiv 13 Sep 2024. ## Builds on (most cited first, 23 of 23) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022) - [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024) - [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021) - [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022) - [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022) - [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023) - [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022) - [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022) - [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020) - [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023) - [Coder Reviewer Reranking for Code Generation](https://mlsanity.com/paper/2211.16490/) (2022) - [Leveraging Automated Unit Tests for Unsupervised Code Translation](https://mlsanity.com/paper/2110.06773/) (2021) - [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023) - [Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks](https://mlsanity.com/paper/2102.02017/) (2021) - [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023) - [Pynguin: Automated Unit Test Generation for Python](https://mlsanity.com/paper/2202.05218/) (2022) - [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022) - [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023) - [Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers](https://mlsanity.com/paper/2009.05634/) (2020) - [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023) - [Using Transfer Learning for Code-Related Tasks](https://mlsanity.com/paper/2206.08574/) (2022) ## Built on by (most cited first, 9 of 9) - [Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation](https://mlsanity.com/paper/2502.14948/) (2025) - [ReCode: Reinforcing Code Generation with Reasoning-Process Rewards](https://mlsanity.com/paper/2508.05170/) (2025) - [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026) - [How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective](https://mlsanity.com/paper/2510.08720/) (2025) - [GiFT: Gibbs Fine-Tuning for Code Generation](https://mlsanity.com/paper/2502.11466/) (2025) - [A Theoretical Analysis of Test-Driven Code Generation](https://mlsanity.com/paper/2602.06098/) (2026) - [Self-Explained Keywords Empower Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15966/) (2024) - [DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern](https://mlsanity.com/paper/2509.24975/) (2025) - [ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation](https://mlsanity.com/paper/2608.04439/) (2026) ## Abstract (arXiv) Selecting the best code solution from multiple generated ones is an essential task in code generation, which can be achieved by using some reliable validators (e.g., developer-written test cases) for assistance. Since reliable test cases are not always available and can be expensive to build in practice, researchers propose to automatically generate test cases to assess code solutions. However, when both code solutions and test cases are plausible and not reliable, selecting the best solution becomes challenging. Although some heuristic strategies have been proposed to tackle this problem, they lack a strong theoretical guarantee and it is still an open question whether an optimal selection strategy exists. Our work contributes in two ways. First, we show that within a Bayesian framework, the optimal selection strategy can be defined based on the posterior probability of the observed passing states between solutions and tests. The problem of identifying the best solution is then framed as an integer programming problem. Second, we propose an efficient approach for approximating this optimal (yet uncomputable) strategy, where the approximation error is bounded by the correctness of prior knowledge. We then incorporate effective prior knowledge to tailor code generation tasks. Both theoretical and empirical studies confirm that existing heuristics are limited in selecting the best solutions with plausible test cases. Our proposed approximated optimal strategy B4 significantly surpasses existing heuristics in selecting code solutions generated by large language models (LLMs) with LLM-generated tests, achieving a relative performance improvement by up to 50% over the strongest heuristic and 246% over the random selection in the most challenging scenarios. Our code is publicly available at https://github.com/ZJU-CTAG/B4.