# Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

arXiv: https://arxiv.org/abs/2109.08780 · HTML: https://mlsanity.com/paper/2109.08780/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Pre-training via Paraphrasing](https://mlsanity.com/paper/2006.15020/) (2020)
- [PyMT5: multi-mode translation of natural language and Python code with transformers](https://mlsanity.com/paper/2010.03150/) (2020)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers](https://mlsanity.com/paper/2009.05634/) (2020)
- [Generating Bug-Fixes Using Pretrained Transformers](https://mlsanity.com/paper/2104.07896/) (2021)

## Built on by (most cited first, 14 of 14)

- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion](https://mlsanity.com/paper/2306.00381/) (2023)
- [DeepPERF: A Deep Learning-Based Approach For Improving Software Performance](https://mlsanity.com/paper/2206.13619/) (2022)
- [Enhancing Project-Specific Code Completion by Inferring Internal API Information](https://mlsanity.com/paper/2507.20888/) (2025)
- [AdaptivePaste: Code Adaptation through Learning Semantics-aware Variable Usage Representations](https://mlsanity.com/paper/2205.11023/) (2022)
- [CodeSSM: Towards State Space Models for Code Understanding](https://mlsanity.com/paper/2505.01475/) (2025)
- [JEMMA: An Extensible Java Dataset for ML4Code Applications](https://mlsanity.com/paper/2212.09132/) (2022)
- [Enabling Global, Human-Centered Explanations for LLMs:From Tokens to Interpretable Code and Test Generation](https://mlsanity.com/paper/2503.16771/) (2025)
- [A Lightweight Framework for Adaptive Retrieval In Code Completion With Critique Model](https://mlsanity.com/paper/2406.10263/) (2024)

## Abstract (arXiv)

Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.