# On the Reliability and Explainability of Language Models for Program Generation

arXiv: https://arxiv.org/abs/2302.09587 · HTML: https://mlsanity.com/paper/2302.09587/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2023.

## Built on by (most cited first, 17 of 17)

- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios](https://mlsanity.com/paper/2410.12468/) (2024)
- [Selective Prompt Anchoring for Code Generation](https://mlsanity.com/paper/2408.09121/) (2024)
- [Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations](https://mlsanity.com/paper/2407.08983/) (2024)
- [An Empirical Study on Capability of Large Language Models in Understanding Code Semantics](https://mlsanity.com/paper/2407.03611/) (2024)
- [Towards Transparent AI: A Survey on Explainable Large Language Models](https://mlsanity.com/paper/2506.21812/) (2025)
- [Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Framework](https://mlsanity.com/paper/2504.13811/) (2025)
- [Looking into Black Box Code Language Models](https://mlsanity.com/paper/2407.04868/) (2024)
- [How Programming Concepts and Neurons Are Shared in Code Language Models](https://mlsanity.com/paper/2506.01074/) (2025)
- [Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?](https://mlsanity.com/paper/2512.19980/) (2025)
- [Towards Transparent AI: A Survey on Explainable Language Models](https://mlsanity.com/paper/2509.21631/) (2025)
- [Icing on the Cake: Automatic Code Summarization at Ericsson](https://mlsanity.com/paper/2408.09735/) (2024)
- [Protect Your Secrets: Understanding and Measuring Data Exposure in VSCode Extensions](https://mlsanity.com/paper/2412.00707/) (2024)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)
- [Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectives](https://mlsanity.com/paper/2503.13793/) (2025)
- [Automated Code Review Using Large Language Models with Symbolic Reasoning](https://mlsanity.com/paper/2507.18476/) (2025)
- [On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study](https://mlsanity.com/paper/2601.23059/) (2026)

## Abstract (arXiv)

Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model-based approaches have been proposed and evaluated on various benchmark datasets, demonstrating promising performance. However, there is still uncertainty about the reliability of these models, particularly their realistic ability to consistently transform code sequences. This raises the question: are these techniques sufficiently trustworthy for automated program generation? Consequently, Further research is needed to understand model logic and assess reliability and explainability. To bridge these research gaps, we conduct a thorough empirical study of eight popular language models on five representative datasets to determine the capabilities and limitations of automated program generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. We discover that state-of-the-art approaches suffer from inappropriate performance evaluation stemming from severe data duplication, causing over-optimistic results. Our explainability analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences. Overall, more rigorous evaluation approaches and benchmarks are critical to enhance the reliability and explainability of automated program generation moving forward. Our findings provide important guidelines for this goal.