# A Comparative Analysis of Large Language Models for Code Documentation Generation

arXiv: https://arxiv.org/abs/2312.10349 · HTML: https://mlsanity.com/paper/2312.10349/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 37 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 6 of 6)

- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Automatic Code Documentation Generation Using GPT-3](https://mlsanity.com/paper/2209.02235/) (2022)

## Built on by (most cited first, 24 of 24)

- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [DocAgent: A Multi-Agent System for Automated Code Documentation Generation](https://mlsanity.com/paper/2504.08725/) (2025)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Migrating Code At Scale With LLMs At Google](https://mlsanity.com/paper/2504.09691/) (2025)
- [Natural Language Outlines for Code: Literate Programming in the LLM Era](https://mlsanity.com/paper/2408.04820/) (2024)
- [Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective](https://mlsanity.com/paper/2508.01443/) (2025)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [Cognitive Agents Powered by Large Language Models for Agile Software Project Management](https://mlsanity.com/paper/2508.16678/) (2025)
- [MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation](https://mlsanity.com/paper/2511.14967/) (2025)
- [Evaluating the Use of LLMs for Documentation to Code Traceability](https://mlsanity.com/paper/2506.16440/) (2025)
- [MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis](https://mlsanity.com/paper/2508.08275/) (2025)
- [Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities](https://mlsanity.com/paper/2409.01382/) (2024)
- [Selective Shot Learning for Code Explanation](https://mlsanity.com/paper/2412.12852/) (2024)
- [Impact of Comments on LLM Comprehension of Legacy Code](https://mlsanity.com/paper/2506.11007/) (2025)
- [DocReward: A Document Reward Model for Structuring and Stylizing](https://mlsanity.com/paper/2510.11391/) (2025)
- [A Systematic Literature Review on the Use of Machine Learning in Software Engineering](https://mlsanity.com/paper/2406.13877/) (2024)
- [ML For Hardware Design Interpretability: Challenges and Opportunities](https://mlsanity.com/paper/2504.08852/) (2025)
- [Impact of LLMs on Team Collaboration in Software Development](https://mlsanity.com/paper/2510.08612/) (2025)
- [Instruction Set Migration at Warehouse Scale](https://mlsanity.com/paper/2510.14928/) (2025)
- [TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance](https://mlsanity.com/paper/2601.18241/) (2026)
- [The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems](https://mlsanity.com/paper/2606.30524/) (2026)
- [Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study](https://mlsanity.com/paper/2609.03702/) (2026)
- [Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer](https://mlsanity.com/paper/2609.31587/) (2026)

## Abstract (arXiv)

This paper presents a comprehensive comparative analysis of Large Language Models (LLMs) for generation of code documentation. Code documentation is an essential part of the software writing process. The paper evaluates models such as GPT-3.5, GPT-4, Bard, Llama2, and Starchat on various parameters like Accuracy, Completeness, Relevance, Understandability, Readability and Time Taken for different levels of code documentation. Our evaluation employs a checklist-based system to minimize subjectivity, providing a more objective assessment. We find that, barring Starchat, all LLMs consistently outperform the original documentation. Notably, closed-source models GPT-3.5, GPT-4, and Bard exhibit superior performance across various parameters compared to open-source/source-available LLMs, namely LLama 2 and StarChat. Considering the time taken for generation, GPT-4 demonstrated the longest duration, followed by Llama2, Bard, with ChatGPT and Starchat having comparable generation times. Additionally, file level documentation had a considerably worse performance across all parameters (except for time taken) as compared to inline and function level documentation.