mlsanityData to 14 Oct 2026

Showing LLM-Generated Code Selectively Based on Confidence of LLMs

arXiv 2410.03234 ↗First posted 4 Oct 2024cs.SE cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive abilities in code generation, but they may generate erroneous programs. Reading a program takes ten times longer than writing it. Showing these erroneous programs to developers will waste developers' energies and introduce security risks to software. To address the above limitations, we propose HonestCoder, a novel LLM-based code generation approach. HonestCoder selectively shows the generated programs to developers based on LLMs' confidence. The confidence provides valuable insights into the correctness of generated programs. To achieve this goal, we propose a novel approach to estimate LLMs' confidence in code generation. It estimates confidence by measuring the multi-modal similarity between LLMs-generated programs. We collect and release a multilingual benchmark named TruthCodeBench, which consists of 2,265 samples and covers two popular programming languages (i.e., Python and Java). We apply HonestCoder to four popular LLMs (e.g., DeepSeek-Coder and Code Llama) and evaluate it on TruthCodeBench. Based on the experiments, we obtain the following insights. (1) HonestCoder can effectively estimate LLMs' confidence and accurately determine the correctness of generated programs. For example, HonestCoder outperforms the state-of-the-art baseline by 27.79% in AUROC and 63.74% in AUCPR. (2) HonestCoder can decrease the number of erroneous programs shown to developers. Compared to eight baselines, it can show more correct programs and fewer erroneous programs to developers. (3) Compared to showing code indiscriminately, HonestCoder only adds slight time overhead (approximately 0.4 seconds per requirement). (4) We discuss future directions to facilitate the application of LLMs in software development. We hope this work can motivate broad discussions about measuring the reliability of LLMs' outputs in performing code-related tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 42 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

24 of 42 references.

References · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  12. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
12 more
  1. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  2. 2023Using an LLM to Help With Code Understandingcited by 228
  3. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  4. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 119
  5. 2018TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generationcited by 88
  6. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  7. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  8. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  9. 2024Calibration and Correctness of Language Models for Codecited by 49
  10. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelitycited by 47
  11. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40
  12. 2023ZC3: Zero-Shot Cross-Language Code Clone Detectioncited by 3

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Incoherence as Oracle-less Measure of Error in LLM-Based Code Generationcited by 6
  2. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  3. 2025Cost-aware LLM-based Online Dataset Annotationcited by 4
  4. 2025Perspective Transition of Large Language Models for Solving Subjective Taskscited by 3