Pragmatic Reasoning improves LLM Code Generation
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
Pragmatic reasoning helps interlocutors infer intended meaning from ambiguous or underspecified messages by considering shared context and counterfactual alternatives. Similar challenges arise in natural language-to-code generation, where user instructions often admit multiple plausible candidate programs. However, direct RSA-style inference is difficult because it requires probability estimation over large spaces of programs and alternative instructions. We propose CodeRSA, an RSA-motivated reranking method that makes pragmatic reasoning tractable through local pragmatic contests among sampled code candidates. CodeRSA constructs candidate-induced alternative instructions and estimates which candidates are most distinctively supported by the original instruction, avoiding global normalization over the full program-instruction space. We evaluate CodeRSA on HumanEval+, MBPP+, and BigCodeBench using four open-weight instruction-following models. CodeRSA achieves the strongest average accuracy in 10 of 12 model-benchmark settings and remains competitive in the remaining cases. Further analyses show that its gains come from combining local pairwise pragmatic comparison with broader global support, suggesting a scalable direction for language-to-code reranking under natural-language uncertainty.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 14 of its 35 references that are in the map. First posted on arXiv 20 Feb 2025.
Builds on
14 of 35 references.
Closest ancestors · 1
Same direction, newest first
- 2024Qwen2.5-Coder Technical Report18 Sep
Other references · 13
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2022CodeT: Code Generation with Generated Testscited by 369
- 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
- 2016Latent Predictor Networks for Code Generationcited by 73
- 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
- 2016Mutual Information and Diverse Decoding Improve Neural Machine Translationcited by 35
- 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisationcited by 31
1 more
- 2024Analysis on LLMs Performance for Code Summarizationcited by 1
Built on by
Citing papers per year: 2025: 2; 2026: 4.
6 papers.
- 2026Value of Information: A Framework for Human-Agent Communicationcited by 10
- 2025On LLMs' Internal Representation of Code Correctnesscited by 5
- 2026CP-SynC: Multi-Agent Zero-Shot Constraint Modeling in MiniZinc with Synthesized Checkerscited by 2
- 2025BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Modelscited by 1
- 2026Context-Augmented Code Generation Using Programming Knowledge Graphscited by 0
- 2026LLMs Infer Cultural Context but Fail to Apply It When Respondingcited by 0
- 2026LLMs Infer Cultural Context but Fail to Apply It When Responding16 Jun 2026
- 2026CP-SynC: Multi-Agent Zero-Shot Constraint Modeling in MiniZinc with Synthesized Checkers3 May 2026
- 2026Context-Augmented Code Generation Using Programming Knowledge Graphs28 Jan 2026
- 2026Value of Information: A Framework for Human-Agent Communication10 Jan 2026
- 2025On LLMs' Internal Representation of Code Correctness8 Dec 2025
- 2025BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models17 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
- 2026Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
- 2026Pseudo2CodeQA: A Benchmark for LLM-Based Structured Algorithmic Reasoning in Code Generation
- 2025CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
- 2025PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code
- 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
- 2025Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs
- 2026Code Is More Than Text: Uncertainty Estimation for Code Generation