# Talking to Itself While Coding: What Makes Comments Help Code Generation?

arXiv: https://arxiv.org/abs/2609.09242 · HTML: https://mlsanity.com/paper/2609.09242/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 43 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!](https://mlsanity.com/paper/2502.07374/) (2025)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository](https://mlsanity.com/paper/2405.01573/) (2024)
- [The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget](https://mlsanity.com/paper/2508.13666/) (2025)
- [Natural Language Outlines for Code: Literate Programming in the LLM Era](https://mlsanity.com/paper/2408.04820/) (2024)
- [Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Execution](https://mlsanity.com/paper/2406.14836/) (2024)
- [Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation](https://mlsanity.com/paper/2408.02502/) (2024)
- [Testing the Effect of Code Documentation on Large Language Model Code Understanding](https://mlsanity.com/paper/2404.03114/) (2024)
- [Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective](https://mlsanity.com/paper/2404.07549/) (2024)
- [Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding](https://mlsanity.com/paper/2505.07768/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) often generate natural-language comments while writing code, and these comments become part of the context used to generate the code that follows. However, it remains unclear which properties of comments affect code-generation performance. We study this question through observational analyses and controlled interventions. On LiveCodeBench, neither comment frequency nor broad comment intent reliably predicts pass@1. We then prefill weaker recipient models with comment blocks written by stronger source models, allowing us to separate comment surface form from the solution content they convey. Comments from source solutions that pass the tests raise recipient pass@1 by 17.2% on average. In contrast, comments describing failed solutions provide no reliable gain, while comments written for a different problem reduce pass@1 by 20.8%. Finally, across a wide range of models and prompt variants, most recipient models show no significant recovery of the external-comment gain, and the best case recovers only 24%. These results show that comments help code generation not merely because they are comments, but because they can provide correct solution content that prompting cannot reliably elicit.