mlsanityData to 7 Oct 2026

What do pre-trained code models know about code?

arXiv 2108.11308 ↗First posted 25 Aug 2021cs.SE cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Pre-trained models of code built on the transformer architecture have performed well on software engineering (SE) tasks such as predictive code generation, code summarization, among others. However, whether the vector representations from these pre-trained models comprehensively encode characteristics of source code well enough to be applicable to a broad spectrum of downstream tasks remains an open question. One way to investigate this is with diagnostic tasks called probes. In this paper, we construct four probing tasks (probing for surface-level, syntactic, structural, and semantic information) for pre-trained code models. We show how probes can be used to identify whether models are deficient in (understanding) certain code properties, characterize different model layers, and get insight into the model sample-efficiency. We probe four models that vary in their expected knowledge of code properties: BERT (pre-trained on English), CodeBERT and CodeBERTa (pre-trained on source code, and natural language documentation), and GraphCodeBERT (pre-trained on source code with dataflow). While GraphCodeBERT performs more consistently overall, we find that BERT performs surprisingly well on some code tasks, which calls for further investigation.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

27 of 35 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  9. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  10. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  11. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 361
15 more
  1. 2020Unsupervised Translation of Programming Languagescited by 262
  2. 2018code2vec: Learning Distributed Representations of Codecited by 249
  3. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  4. 2017Learning to Represent Programs with Graphscited by 219
  5. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  6. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  7. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  8. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  9. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  10. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  11. 2018Generative Code Modeling with Graphscited by 59
  12. 2020LambdaNet: Probabilistic Type Inference using Graph Neural Networkscited by 44
  13. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  14. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  15. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 12; 2024: 9; 2025: 5; 2026: 3.

36 papers.

Sort:

By area:

  1. 2022Probing Pretrained Models of Source Codecited by 26
  2. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20
  3. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  4. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  5. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16
  6. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  7. 2022Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluationcited by 14
  8. 2023From Copilot to Pilot: Towards AI Supported Software Developmentcited by 14
  9. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 13
  10. 2022An Exploratory Study on Code Attention in BERTcited by 12
  11. 2022Benchmarking Language Models for Code Syntax Understandingcited by 8
  12. 2024Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Studycited by 7
24 more
  1. 2022CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structurecited by 6
  2. 2023The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classificationcited by 5
  3. 2024A Critical Study of What Code-LLMs (Do Not) Learncited by 4
  4. 2024Do Current Language Models Support Code Intelligence for R Programming Language?cited by 4
  5. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
  6. 2023Towards Understanding What Code Language Models Learnedcited by 3
  7. 2023The Behavior of Large Language Models When Prompted to Generate Code Explanationscited by 3
  8. 2024Structure-aware Fine-tuning for Code Pre-trained Modelscited by 3
  9. 2024Personality-Guided Code Generation Using Large Language Modelscited by 3
  10. 2023xASTNN: Improved Code Representations for Industrial Practicecited by 2
  11. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2
  12. 2024LecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERTcited by 2
  13. 2022Learning to Learn to Predict Performance Regressions in Production at Metacited by 1
  14. 2023Redundancy and Concept Analysis for Code-trained Language Modelscited by 1
  15. 2024Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidancecited by 1
  16. 2024DeepCodeProbe: Towards Understanding What Models Trained on Code Learncited by 1
  17. 2025Analyzing Latent Concepts in Code Language Modelscited by 1
  18. 2025Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agentcited by 1
  19. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  20. 2024Advanced Detection of Source Code Clones via an Ensemble of Unsupervised Similarity Measurescited by 0
  21. 2025LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Modelscited by 0
  22. 2025LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Codecited by 0
  23. 2026Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generationcited by 0
  24. 2026Latent Programming Horizons in Coding Agentscited by 0