mlsanityData to 7 Oct 2026

CoTexT: Multi-task Learning with Code-Text Transformer

arXiv 2105.08645 ↗First posted 18 May 2021cs.AI cs.PLcited by 61

Not on the map. Why?

Abstract (arXiv)

We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both "bimodal" and "unimodal" data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

21 of 25 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
9 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  5. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  6. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  8. 2020Unsupervised Translation of Programming Languagescited by 262
  9. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 23; 2023: 19; 2024: 8; 2025: 5; 2026: 2.

61 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  5. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  6. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  7. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  8. 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
  9. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  10. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  11. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 44
  12. 2021Multilingual training for Software Engineeringcited by 43
38 more
  1. 2023RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repaircited by 43
  2. 2022ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generationcited by 37
  3. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 33
  4. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  5. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  6. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
  7. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  8. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 20
  9. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  10. 2024Analyzing the Performance of Large Language Models on Code Summarizationcited by 19
  11. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  12. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEscited by 17
  13. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  14. 2022CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Trainingcited by 15
  15. 2022An Extractive-and-Abstractive Framework for Source Code Summarizationcited by 15
  16. 2022A Survey on Pretrained Language Models for Neural Code Intelligencecited by 13
  17. 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generatorscited by 10
  18. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  19. 2022Fix Bugs with Transformer through a Neural-Symbolic Edit Grammarcited by 8
  20. 2023TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skillscited by 8
  21. 2024CommitBench: A Benchmark for Commit Message Generationcited by 8
  22. 2022Pre-Training Representations of Binary Code Using Contrastive Learningcited by 6
  23. 2024RepoGenReflex: Enhancing Repository-Level Code Completion with Verbal Reinforcement and Retrieval-Augmented Generationcited by 6
  24. 2022Learning code summarization from a small and local datasetcited by 5
  25. 2023The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classificationcited by 5
  26. 2024ChatGPT Incorrectness Detection in Software Reviewscited by 5
  27. 2022Enriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translationcited by 4
  28. 2023JaCoText: A Pretrained Model for Java Code-Text Generationcited by 4
  29. 2024Does Self-Attention Need Separate Weights in Transformers?cited by 4
  30. 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
  31. 2022Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelinescited by 3
  32. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  33. 2025SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketchescited by 3
  34. 2022Semantic-Preserving Adversarial Code Comprehensioncited by 2
  35. 2023A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Textcited by 2
  36. 2025Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detectioncited by 2
  37. 2022Addressing Leakage in Self-Supervised Contextualized Code Retrievalcited by 1
  38. 2023Knowledge Transfer for Pseudo-code Generation from Low Resource Programming Languagecited by 1

50 most cited of 61 All 61 (JSON)