mlsanityData to 7 Oct 2026

Contrastive Code Representation Learning

arXiv 2007.04973 ↗First posted 9 Jul 2020cs.LG cs.AI cs.PLcited by 76

Not on the map. Why?

Abstract (arXiv)

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like summarizing code in English, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based BERT model is sensitive to source code edits, even when the edits preserve semantics. We propose ContraCode: a contrastive pre-training task that learns code functionality, not form. ContraCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pre-training improves JavaScript summarization and TypeScript type inference accuracy by 2% to 13%. We also propose a new zero-shot JavaScript code clone detection dataset, showing that ContraCode is both more robust and semantically meaningful. On it, we outperform RoBERTa by 39% AUROC in an adversarial setting and up to 5% on natural code.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 22; 2023: 15; 2024: 12; 2025: 9; 2026: 8.

76 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  4. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  5. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  6. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  7. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  8. 2021Code Structure Guided Transformer for Source Code Summarizationcited by 29
  9. 2021Bridging Pre-trained Models and Downstream Tasks for Source Code Understandingcited by 27
  10. 2021How could Neural Networks understand Programs?cited by 26
  11. 2024Code Representation Learning At Scalecited by 26
  12. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
38 more
  1. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  2. 2022CoCoSoDa: Effective Contrastive Learning for Code Searchcited by 21
  3. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  4. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 20
  5. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  6. 2022Assemble Foundation Models for Automatic Code Summarizationcited by 17
  7. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generationcited by 16
  8. 2022On Distribution Shift in Learning-based Bug Detectorscited by 15
  9. 2022CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Trainingcited by 15
  10. 2022A Survey on Pretrained Language Models for Neural Code Intelligencecited by 13
  11. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 13
  12. 2024ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Searchcited by 12
  13. 2021VarCLR: Variable Semantic Representation Pre-training via Contrastive Learningcited by 11
  14. 2023Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separatelycited by 9
  15. 2021ProtoTransformer: A Meta-Learning Approach to Providing Student Feedbackcited by 8
  16. 2022Exploring Representation-Level Augmentation for Code Searchcited by 8
  17. 2022Poison Attack and Defense on Deep Source Code Processing Modelscited by 8
  18. 2022Soft-Labeled Contrastive Pre-training for Function-level Code Representationcited by 7
  19. 2022CommitBART: A Large Pre-trained Model for GitHub Commitscited by 6
  20. 2022Pre-Training Representations of Binary Code Using Contrastive Learningcited by 6
  21. 2023An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graphcited by 5
  22. 2025The Code Barrier: What LLMs Actually Understand?cited by 5
  23. 2025SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrievalcited by 5
  24. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  25. 2023TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformationcited by 4
  26. 2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
  27. 2024CodeFort: Robust Training for Code Generation Modelscited by 4
  28. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  29. 2021Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomizationcited by 3
  30. 2022NS3: Neuro-Symbolic Semantic Code Searchcited by 3
  31. 2022Evaluation of Contrastive Learning with Various Code Representations for Code Clone Detectioncited by 3
  32. 2022Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelinescited by 3
  33. 2023Neural Machine Translation for Code Generationcited by 3
  34. 2023Towards Understanding What Code Language Models Learnedcited by 3
  35. 2023ZC3: Zero-Shot Cross-Language Code Clone Detectioncited by 3
  36. 2024Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translationcited by 3
  37. 2024Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Studycited by 3
  38. 2024BLens: Contrastive Captioning of Binary Functions using Ensemble Embeddingcited by 3

50 most cited of 76 All 76 (JSON)