mlsanityData to 7 Oct 2026

Probing Pretrained Models of Source Code

arXiv 2202.08975 ↗First posted 16 Feb 2022cs.SE cs.CL cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Deep learning models are widely used for solving challenging code processing tasks, such as code generation or code summarization. Traditionally, a specific model architecture was carefully built to solve a particular code processing task. However, recently general pretrained models such as CodeBERT or CodeT5 have been shown to outperform task-specific models in many applications. While pretrained models are known to learn complex patterns from data, they may fail to understand some properties of source code. To test diverse aspects of code understanding, we introduce a set of diagnosting probing tasks. We show that pretrained models of code indeed contain information about code syntactic structure and correctness, the notions of identifiers, data flow and namespaces, and natural language naming. We also investigate how probing results are affected by using code-specific pretraining objectives, varying the model size, or finetuning.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

26 of 52 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
14 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  6. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  7. 2020Unsupervised Translation of Programming Languagescited by 262
  8. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  9. 2020Improved Code Summarization via a Graph Neural Networkcited by 79
  10. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  11. 2021Discourse Probing of Pretrained Language Modelscited by 38
  12. 2021What do pre-trained code models know about code?cited by 36
  13. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  14. 2020Empirical Study of Transformers for Source Codecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 6; 2025: 6; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  2. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20
  3. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  4. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  5. 2022Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacitiescited by 9
  6. 2023Evaluating and Explaining Large Language Models for Code Using Syntactic Structurescited by 9
  7. 2024AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generationcited by 8
  8. 2022CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structurecited by 6
  9. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  10. 2022SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Modelscited by 5
  11. 2023Substance or Style: What Does Your Image Embedding Know?cited by 5
  12. 2024Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Languagecited by 5
14 more
  1. 2024A Critical Study of What Code-LLMs (Do Not) Learncited by 4
  2. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  3. 2023Towards Understanding What Code Language Models Learnedcited by 3
  4. 2023Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?cited by 3
  5. 2024Structure-aware Fine-tuning for Code Pre-trained Modelscited by 3
  6. 2025Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoderscited by 3
  7. 2025Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?cited by 3
  8. 2025Localized Calibrated Uncertainty in Code Language Modelscited by 3
  9. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2
  10. 2025Analyzing Latent Concepts in Code Language Modelscited by 1
  11. 2026Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streamscited by 1
  12. 2025Toward Neurosymbolic Program Comprehensioncited by 0
  13. 2026Latent Programming Horizons in Coding Agentscited by 0
  14. 2026TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Modelscited by 0