# Semantic Similarity Loss for Neural Source Code Summarization arXiv: https://arxiv.org/abs/2308.07429 ยท HTML: https://mlsanity.com/paper/2308.07429/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 40 of its 81 references that are in the map. - First posted on arXiv 14 Aug 2023. ## Builds on (most cited first, 30 of 40) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015) - [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018) - [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022) - [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018) - [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023) - [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018) - [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020) - [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022) - [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018) - [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023) - [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020) - [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019) - [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022) - [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020) - [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019) - [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018) - [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023) - [Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks](https://mlsanity.com/paper/2102.02017/) (2021) - [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021) - [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019) - [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://mlsanity.com/paper/1711.05732/) (2017) - [Automatic Code Documentation Generation Using GPT-3](https://mlsanity.com/paper/2209.02235/) (2022) - [Improved Automatic Summarization of Subroutines via Attention to File Context](https://mlsanity.com/paper/2004.04881/) (2020) - [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023) - [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021) ## Built on by (most cited first, 5 of 5) - [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023) - [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025) - [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023) - [Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks](https://mlsanity.com/paper/2504.19444/) (2025) - [AI-Mediated Code Comment Improvement](https://mlsanity.com/paper/2505.09021/) (2025) ## Abstract (arXiv) This paper presents a procedure for and evaluation of using a semantic similarity metric as a loss function for neural source code summarization. Code summarization is the task of writing natural language descriptions of source code. Neural code summarization refers to automated techniques for generating these descriptions using neural networks. Almost all current approaches involve neural networks as either standalone models or as part of a pretrained large language models e.g., GPT, Codex, LLaMA. Yet almost all also use a categorical cross-entropy (CCE) loss function for network optimization. Two problems with CCE are that 1) it computes loss over each word prediction one-at-a-time, rather than evaluating a whole sentence, and 2) it requires a perfect prediction, leaving no room for partial credit for synonyms. In this paper, we extend our previous work on semantic similarity metrics to show a procedure for using semantic similarity as a loss function to alleviate this problem, and we evaluate this procedure in several settings in both metrics-driven and human studies. In essence, we propose to use a semantic similarity metric to calculate loss over the whole output sentence prediction per training batch, rather than just loss for each word. We also propose to combine our loss with CCE for each word, which streamlines the training process compared to baselines. We evaluate our approach over several baselines and report improvement in the vast majority of conditions.