mlsanityData to 7 Oct 2026

TreeBERT: A Tree-Based Pre-Trained Model for Programming Language

arXiv 2105.12485 ↗First posted 26 May 2021cs.LG cs.PLcited by 40

Not on the map. Why?

Abstract (arXiv)

Source code can be parsed into the abstract syntax tree (AST) based on defined syntax rules. However, in pre-training, little work has considered the incorporation of tree structure into the learning process. In this paper, we present TreeBERT, a tree-based pre-trained model for improving programming language-oriented generation tasks. To utilize tree structure, TreeBERT represents the AST corresponding to the code as a set of composition paths and introduces node position embedding. The model is trained by tree masked language modeling (TMLM) and node order prediction (NOP) with a hybrid objective. TMLM uses a novel masking strategy designed according to the tree's characteristics to help the model understand the AST and infer the missing semantics of the AST. With NOP, TreeBERT extracts the syntactical structure by learning the order constraints of nodes in AST. We pre-trained TreeBERT on datasets covering multiple programming languages. On code summarization and code documentation tasks, TreeBERT outperforms other pre-trained models and state-of-the-art models designed for these tasks. Furthermore, TreeBERT performs well when transferred to the pre-trained unseen programming language.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

24 of 33 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2017Convolutional Sequence to Sequence Learningcited by 550
  12. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
12 more
  1. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  2. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  3. 2020Unsupervised Translation of Programming Languagescited by 262
  4. 2018code2vec: Learning Distributed Representations of Codecited by 249
  5. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  6. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  7. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  8. 2020Improved Code Summarization via a Graph Neural Networkcited by 79
  9. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  10. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  11. 2018Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networkscited by 36
  12. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 9; 2024: 9; 2025: 10; 2026: 1.

40 papers.

Sort:

By area:

  1. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  2. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  3. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  4. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  5. 2022On the Transferability of Pre-trained Language Models for Low-Resource Programming Languagescited by 30
  6. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  7. 2024Code Representation Learning At Scalecited by 26
  8. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  9. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 20
  10. 2023Multi-target Backdoor Attacks for Code Pre-trained Modelscited by 19
  11. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  12. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
28 more
  1. 2022CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Trainingcited by 15
  2. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  3. 2024GALLa: Graph Aligned Large Language Models for Improved Source Code Understandingcited by 7
  4. 2022CommitBART: A Large Pre-trained Model for GitHub Commitscited by 6
  5. 2022Learning Program Representations with a Tree-Structured Transformercited by 6
  6. 2023An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graphcited by 5
  7. 2023Abstract Syntax Tree for Programming Language Understanding and Representation: How Far Are We?cited by 5
  8. 2025Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?cited by 5
  9. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  10. 2024CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generationcited by 4
  11. 2022Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelinescited by 3
  12. 2023Neural Machine Translation for Code Generationcited by 3
  13. 2024Structured Code Representations Enable Data-Efficient Adaptation of Code Language Modelscited by 3
  14. 2024Towards Better Code Understanding in Decoder-Only Models with Contrastive Learningcited by 3
  15. 2024Code LLMs: A Taxonomy-based Surveycited by 3
  16. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  17. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  18. 2026Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migrationcited by 3
  19. 2023APIContext2Com: Code Comment Generation by Incorporating Pre-Defined API Documentationcited by 2
  20. 2023Neuro-symbolic Zero-Shot Code Cloning with Cross-Language Intermediate Representationcited by 2
  21. 2025Speculative Decoding for Verilog: Speed and Quality, All in Onecited by 2
  22. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  23. 2024Text-to-Code Generation with Modality-relative Pre-trainingcited by 1
  24. 2024Analysing the Behaviour of Tree-Based Neural Networks in Regression Taskscited by 0
  25. 2024CogNarr Ecosystem: Facilitating Group Cognition at Scalecited by 0
  26. 2025LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Modelscited by 0
  27. 2025Interpreting Performance Profiles with Deep Learningcited by 0
  28. 2025Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimizationcited by 0