# Assertion-Aware Test Code Summarization with Large Language Models

arXiv: https://arxiv.org/abs/2511.06227 · HTML: https://mlsanity.com/paper/2511.06227/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories](https://mlsanity.com/paper/2412.17298/) (2024)
- [Can Large Language Models Serve as Evaluators for Code Summarization?](https://mlsanity.com/paper/2412.01333/) (2024)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [PromptSet: A Programmer's Prompting Dataset](https://mlsanity.com/paper/2402.16932/) (2024)
- [ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code](https://mlsanity.com/paper/2411.16927/) (2024)
- [Test Smell Detection Tools: A Systematic Mapping Study](https://mlsanity.com/paper/2104.14640/) (2021)
- [Icing on the Cake: Automatic Code Summarization at Ericsson](https://mlsanity.com/paper/2408.09735/) (2024)
- [An Exploratory Study on the Usage and Readability of Messages Within Assertion Methods of Test Cases](https://mlsanity.com/paper/2303.00169/) (2023)
- [Assertion Messages with Large Language Models (LLMs) for Code](https://mlsanity.com/paper/2509.19673/) (2025)

## Abstract (arXiv)

Unit tests often lack concise summaries that convey test intent, especially in auto-generated or poorly documented codebases. Large Language Models (LLMs) offer a promising solution, but their effectiveness depends heavily on how they are prompted. Unlike generic code summarization, test-code summarization poses distinct challenges because test methods validate expected behavior through assertions rather than implementing functionality. This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries and conducts a controlled ablation study to investigate how test code-related components-such as the method under test (MUT), assertion messages, and assertion semantics-affect the performance of LLM-generated test summaries. We evaluate four code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) across seven prompt configurations using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic similarity (BERTScore), and LLM-based evaluation. Results show that prompting with assertion semantics improves summary quality by an average of 0.10 points (2.3%) over full MUT context (4.45 vs. 4.35) while requiring fewer input tokens. Codex and Qwen-Coder achieve the highest alignment with human-written summaries, while DeepSeek underperforms despite high lexical overlap. The replication package is publicly available at https://doi.org/10. 5281/zenodo.17067550