# Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization

arXiv: https://arxiv.org/abs/2305.09773 · HTML: https://mlsanity.com/paper/2305.09773/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 19 of 19)

- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features](https://mlsanity.com/paper/1810.10974/) (2018)
- [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019)
- [Improved Automatic Summarization of Subroutines via Attention to File Context](https://mlsanity.com/paper/2004.04881/) (2020)
- [Project-Level Encoding for Neural Source Code Summarization of Subroutines](https://mlsanity.com/paper/2103.11599/) (2021)
- [Top-down Visual Saliency Guided by Captions](https://mlsanity.com/paper/1612.07360/) (2016)
- [Using Human Brain Activity to Guide Machine Learning](https://mlsanity.com/paper/1703.05463/) (2017)
- [Semantic Similarity Metrics for Evaluating Source Code Summarization](https://mlsanity.com/paper/2204.01632/) (2022)
- [Action Word Prediction for Neural Source Code Summarization](https://mlsanity.com/paper/2101.02742/) (2021)
- [Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017](https://mlsanity.com/paper/1711.08378/) (2017)
- [Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters](https://mlsanity.com/paper/1903.03358/) (2019)
- [gazel: Supporting Source Code Edits in Eye-Tracking Studies](https://mlsanity.com/paper/2106.10563/) (2021)

## Built on by (most cited first, 8 of 8)

- [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)
- [EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention](https://mlsanity.com/paper/2508.16771/) (2025)
- [Revisiting File Context for Source Code Summarization](https://mlsanity.com/paper/2309.02326/) (2023)
- [Enhancing Code LLM Training with Programmer Attention](https://mlsanity.com/paper/2503.14936/) (2025)
- [EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization](https://mlsanity.com/paper/2602.22368/) (2026)
- [Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study](https://mlsanity.com/paper/2609.03702/) (2026)

## Abstract (arXiv)

Neural source code summarization is the task of generating natural language descriptions of source code behavior using neural networks. A fundamental component of most neural models is an attention mechanism. The attention mechanism learns to connect features in source code to specific words to use when generating natural language descriptions. Humans also pay attention to some features in code more than others. This human attention reflects experience and high-level cognition well beyond the capability of any current neural model. In this paper, we use data from published eye-tracking experiments to create a model of this human attention. The model predicts which words in source code are the most important for code summarization. Next, we augment a baseline neural code summarization approach using our model of human attention. We observe an improvement in prediction performance of the augmented approach in line with other bio-inspired neural models.