# Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization

arXiv: https://arxiv.org/abs/2305.11074 · HTML: https://mlsanity.com/paper/2305.11074/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RAG for code, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 18 May 2023.

## Placed in

- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees](https://mlsanity.com/paper/2108.12987/) (2021)
- [Code Summarization with Structure-induced Transformer](https://mlsanity.com/paper/2012.14710/) (2020)
- [Tree-to-Sequence Attentional Neural Machine Translation](https://mlsanity.com/paper/1603.06075/) (2016)
- [Automatic Source Code Summarization with Extended Tree-LSTM](https://mlsanity.com/paper/1906.08094/) (2019)
- [Learning Kernel-Smoothed Machine Translation with Retrieved Examples](https://mlsanity.com/paper/2109.09991/) (2021)
- [A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes](https://mlsanity.com/paper/1704.04856/) (2017)
- [Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/2109.06604/) (2021)
- [Automatic Generation of Text Descriptive Comments for Code Blocks](https://mlsanity.com/paper/1808.06880/) (2018)
- [Source Code Summarization with Structural Relative Position Guided Transformer](https://mlsanity.com/paper/2202.06521/) (2022)

## Built on by (most cited first, 6 of 6)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [ReZG: Retrieval-Augmented Zero-Shot Counter Narrative Generation for Hate Speech](https://mlsanity.com/paper/2310.05650/) (2023)
- [CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code](https://mlsanity.com/paper/2310.16853/) (2023)
- [ModiGen: A Large Language Model-Based Workflow for Multi-Task Modelica Code Generation](https://mlsanity.com/paper/2503.18460/) (2025)
- [ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?](https://mlsanity.com/paper/2509.09192/) (2025)
- [Grounded Skill Synthesis from Code at Scale for Agentic Intelligence](https://mlsanity.com/paper/2609.05571/) (2026)

## Abstract (arXiv)

Automatically generating human-readable text describing the functionality of a program is the intent of source code summarization. Although neural language models achieve significant performance in this field, they are limited by their inability to access external knowledge. To address this limitation, an emerging trend is combining neural models with external knowledge through retrieval methods. Previous methods have relied on the sentence-level retrieval paradigm on the encoder side. However, this paradigm is coarse-grained, noise-filled and cannot directly take advantage of the high-quality retrieved summary tokens on the decoder side. In this paper, we propose a fine-grained Token-level retrieval-augmented mechanism (Tram) on the decoder side rather than the encoder side to enhance the performance of neural models and produce more low-frequency tokens in generating summaries. Furthermore, to overcome the challenge of token-level retrieval in capturing contextual code semantics, we also propose integrating code semantics into individual summary tokens. The results of extensive experiments and human evaluation show that our token-level retrieval-augmented approach significantly improves performance and is more interpretable.