mlsanityData to 7 Oct 2026

Empirical Study of Transformers for Source Code

arXiv 2010.07987 ↗First posted 15 Oct 2020cs.LG cs.CL cs.SEcited by 13

Not on the map. Why?

Abstract (arXiv)

Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. In contrast to natural language, source code is strictly structured, i.e., it follows the syntax of the programming language. Several recent works develop Transformer modifications for capturing syntactic information in source code. The drawback of these works is that they do not compare to each other and consider different tasks. In this work, we conduct a thorough empirical study of the capabilities of Transformers to utilize syntactic information in different tasks. We consider three tasks (code completion, function naming and bug fixing) and re-implement different syntax-capturing modifications in a unified framework. We show that Transformers are able to make meaningful predictions based purely on syntactic information and underline the best practices of taking the syntactic information into account for improving the performance of the model.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 8; 2024: 0; 2025: 1.

13 papers.

Sort:

By area:

  1. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  2. 2022Probing Pretrained Models of Source Codecited by 26
  3. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  4. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  5. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  6. 2022Benchmarking Language Models for Code Syntax Understandingcited by 8
  7. 2023Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Studycited by 7
  8. 2023Boosting Source Code Learning with Text-Oriented Data Augmentation: An Empirical Studycited by 5
  9. 2023A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Taskscited by 5
  10. 2023What are the Machine Learning best practices reported by practitioners on Stack Exchange?cited by 3
  11. 2025LLM Agents for Automated Dependency Upgradescited by 1
  12. 2022Neural Transition-based Parsing of Library Deprecationscited by 0
1 more
  1. 2023Towards Code Generation from BDD Test Case Specifications: A Visioncited by 0