mlsanityData to 14 Oct 2026

On the Impact of Language Selection for Training and Evaluating Programming Language Models

arXiv 2308.13354 ↗First posted 25 Aug 2023cs.SE cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable progress made in this domain not only applies to natural language tasks but also extends to the domain of programming languages. Despite the ability of these models to learn from multiple languages, evaluations typically focus on particular combinations of the same languages. In this study, we evaluate the similarity of programming languages by analyzing their representations using a CodeBERT-based model. Our experiments reveal that token representation in languages such as C++, Python, and Java exhibit proximity to one another, whereas the same tokens in languages such as Mathematica and R display significant dissimilarity. Our findings suggest that this phenomenon can potentially result in performance challenges when dealing with diverse languages. Thus, we recommend using our similarity measure to select a diverse set of programming languages when training and evaluating future models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 23 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

21 of 23 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  7. 2023StarCoder: may the source be with you!cited by 799
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  9. 2019How multilingual is Multilingual BERT?cited by 645
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  11. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  12. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
9 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  2. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  3. 2023SantaCoder: don't reach for the stars!cited by 170
  4. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  5. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  6. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 50
  7. 2022On the Transferability of Pre-trained Language Models for Low-Resource Programming Languagescited by 30
  8. 2023Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binariescited by 26
  9. 2023Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Studycited by 8

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Language Models for Code Completion: A Practical Evaluationcited by 39
  2. 2024Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Studycited by 7
  3. 2024A Transformer-Based Approach for Smart Invocation of Automatic Code Completioncited by 5
  4. 2025A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metricscited by 2