mlsanityData to 13 Oct 2026

CommitBART: A Large Pre-trained Model for GitHub Commits

arXiv 2208.08100 ↗First posted 17 Aug 2022cs.SE cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

GitHub commits, which record the code changes with natural language messages for description, play a critical role for software developers to comprehend the software evolution. To promote the development of the open-source software community, we collect a commit benchmark including over 7.99 million commits across 7 programming languages. Based on this benchmark, we present CommitBART, a large pre-trained encoder-decoder Transformer model for GitHub commits. The model is pre-trained by three categories (i.e., denoising objectives, cross-modal generation and contrastive learning) for six pre-training tasks to learn commit fragment representations. Furthermore, we unify a ``commit intelligence'' framework with one understanding task and three generation tasks for commits. The comprehensive experiments on these tasks demonstrate that CommitBARTsignificantly outperforms previous pre-trained works for code. Further analysis also reveals each pre-training task enhances the model performance.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

32 of 36 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,841
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,304
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 905
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
  11. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 366
20 more
  1. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 360
  2. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  3. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 125
  4. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 94
  5. 2020Contrastive Code Representation Learningcited by 76
  6. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  7. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  8. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  9. 2021Towards Automating Code Review Activitiescited by 54
  10. 2017Automatically Generating Commit Messages from Diffs using Neural Machine Translationcited by 42
  11. 2021TreeBERT: A Tree-Based Pre-Trained Model for Programming Languagecited by 41
  12. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  13. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  14. 2019ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Rankingcited by 25
  15. 2021On Multi-Modal Learning of Editing Source Codecited by 25
  16. 2021On the Evaluation of Commit Message Generation Models: An Experimental Studycited by 20
  17. 2021SPI: Automated Identification of Security Patches via Commitscited by 19
  18. 2021CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Modelcited by 16
  19. 2021PatchRNN: A Deep Learning-Based System for Security Patch Identificationcited by 13
  20. 2022Enhancing Security Patch Identification by Capturing Structures in Commitscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 0; 2026: 2.

6 papers.

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  2. 2023Multi-target Backdoor Attacks for Code Pre-trained Modelscited by 23
  3. 2022Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacitiescited by 9
  4. 2024Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learningcited by 4
  5. 2026Learning to Commit: Generating Organic Pull Requests via Online Repository Memorycited by 3
  6. 2026Stylometric Defenses Against Author Impersonation in Software Repositoriescited by 0