mlsanityData to 7 Oct 2026

Assemble Foundation Models for Automatic Code Summarization

arXiv 2201.05222 ↗First posted 13 Jan 2022cs.SE cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Automatic code summarization is beneficial to daily software development since it could help reduce the requirement of manual writing. Currently, artificial intelligence is undergoing a paradigm shift. The foundation models pretrained on massive data and finetuned to downstream tasks surpass specially customized models. This trend inspired us to consider reusing foundation models instead of learning from scratch. Thereby, we propose a flexible and robust approach for automatic code summarization, based on neural models. We assemble available foundation models, such as CodeBERT and GPT-2, into a single neural model named AdaMo. Moreover, we utilize Gaussian noise as the simulation of contextual information to optimize the latent representation. Furthermore, we introduce two adaptive schemes from the perspective of knowledge transfer, namely continuous pretraining and intermediate finetuning, and design intermediate stage tasks for general sequence-to-sequence learning. Finally, we evaluate AdaMo against a benchmark dataset for code summarization, by comparing it with state-of-the-art models.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 55 of its 76 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

55 of 76 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
38 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  3. 2018Self-Attention with Relative Position Representationscited by 1,112
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2015Pointer Networkscited by 882
  7. 2015Gated Graph Sequence Neural Networkscited by 792
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  11. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  13. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  14. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  15. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  16. 2017Learning to Represent Programs with Graphscited by 219
  17. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  18. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  19. 2018Bottom-Up Abstractive Summarizationcited by 177
  20. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  21. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  22. 2016Modeling Coverage for Neural Machine Translationcited by 110
  23. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
  24. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  25. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  26. 2020Contrastive Code Representation Learningcited by 76
  27. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
  28. 2017Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generationcited by 61
  29. 2019Knowledge Guided Text Retrieval and Reading for Open Domain Question Answeringcited by 59
  30. 2020Felix: Flexible Text Editing Through Tagging and Insertioncited by 58
  31. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  32. 2020Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translationcited by 34
  33. 2020Code Summarization with Structure-induced Transformercited by 29
  34. 2018A General Path-Based Representation for Predicting Program Propertiescited by 28
  35. 2020Code to Comment "Translation": Data, Metrics, Baselining & Evaluationcited by 28
  36. 2020On Losses for Modern Language Modelscited by 21
  37. 2021Improving Code Summarization with Block-wise Abstract Syntax Tree Splittingcited by 19
  38. 2019Automatic Code Summarization: A Systematic Literature Reviewcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 10; 2025: 0; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  3. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  4. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  5. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  6. 2024Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical studycited by 19
  7. 2023HPC-Coder: Modeling Parallel Programs using Large Language Modelscited by 11
  8. 2024RepoMasterEval: Evaluating Code Completion via Real-World Repositoriescited by 10
  9. 2024ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarizationcited by 6
  10. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Codecited by 6
  11. 2023Semantic Similarity Loss for Neural Source Code Summarizationcited by 5
  12. 2023Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generationcited by 4
5 more
  1. 2024Detecting Buggy Contracts via Smart Testingcited by 3
  2. 2024On the Compression of Language Models for Code: An Empirical Study on CodeBERTcited by 3
  3. 2022Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executablescited by 2
  4. 2022Unveiling Project-Specific Bias in Neural Code Modelscited by 1
  5. 2024Unlocking Insights: Semantic Search in Jupyter Notebookscited by 0