# Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation

arXiv: https://arxiv.org/abs/2411.14971 · HTML: https://mlsanity.com/paper/2411.14971/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 64 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 11 of 11)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization](https://mlsanity.com/paper/2312.15475/) (2023)
- [Automatic Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/1909.04352/) (2019)
- [Towards Neural Decompilation](https://mlsanity.com/paper/1905.08325/) (2019)
- [Verified Code Transpilation with LLMs](https://mlsanity.com/paper/2406.03003/) (2024)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [A Neural-based Program Decompiler](https://mlsanity.com/paper/1906.12029/) (2019)

## Built on by (most cited first, 6 of 6)

- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [Impact of Comments on LLM Comprehension of Legacy Code](https://mlsanity.com/paper/2506.11007/) (2025)
- [Application Modernization with LLMs: Addressing Core Challenges in Reliability, Security, and Quality](https://mlsanity.com/paper/2506.10984/) (2025)
- [EvoGraph: Hybrid Directed Graph Evolution toward Software 3.0](https://mlsanity.com/paper/2508.05199/) (2025)
- [CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernization](https://mlsanity.com/paper/2609.11332/) (2026)

## Abstract (arXiv)

Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems.