mlsanityData to 14 Oct 2026

Do Current Language Models Support Code Intelligence for R Programming Language?

arXiv 2410.07793 ↗First posted 10 Oct 2024cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advancements in developing Pre-trained Language Models for Code (Code-PLMs) have urged many areas of Software Engineering (SE) and brought breakthrough results for many SE tasks. Though these models have achieved the state-of-the-art performance for SE tasks for many popular programming languages, such as Java and Python, the Scientific Software and its related languages like R programming language have rarely benefited or even been evaluated with the Code-PLMs. Research has shown that R has many differences with other programming languages and requires specific techniques. In this study, we provide the first insights for code intelligence for R. For this purpose, we collect and open source an R dataset, and evaluate Code-PLMs for the two tasks of code summarization and method name prediction using several settings and strategies, including the differences in two R styles, Tidy-verse and Base R. Our results demonstrate that the studied models have experienced varying degrees of performance degradation when processing R programming language code, which is supported by human evaluation. Additionally, not all models show performance improvement in R-specific tasks even after multi-language fine-tuning. The dual syntax paradigms in R significantly impact the models' performance, particularly in code summarization tasks. Furthermore, the project-specific context inherent in R codebases significantly impacts the performance when attempting cross-project training.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 46 of its 96 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

46 of 96 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
34 more
  1. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  2. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2018code2vec: Learning Distributed Representations of Codecited by 250
  5. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  6. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  7. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  8. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  9. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  10. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  11. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  12. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 91
  13. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  14. 2019Recommendations for Datasets for Source Code Summarizationcited by 46
  15. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  16. 2021Multilingual training for Software Engineeringcited by 43
  17. 2021What do pre-trained code models know about code?cited by 36
  18. 2023The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Pythoncited by 29
  19. 2023An Empirical Comparison of Pre-Trained Models of Source Codecited by 25
  20. 2019Automatic Code Summarization: A Systematic Literature Reviewcited by 18
  21. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16
  22. 2021HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebookscited by 6
  23. 2022Bug Analysis in Jupyter Notebook Projects: An Empirical Studycited by 5
  24. 2020CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysiscited by 3
  25. 2020Software Engineering Practices for Scientific Software Development: A Systematic Mapping Studycited by 3
  26. 2021ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cellscited by 3
  27. 2022Expansion and evolution of the R programming languagecited by 3
  28. 2023Characterizing Bugs in Python and R Data Analytics Programscited by 3
  29. 2023Towards Understanding What Code Language Models Learnedcited by 3
  30. 2013Possible Directions for Improving Dependency Versioning in Rcited by 2
  31. 2023APIContext2Com: Code Comment Generation by Incorporating Pre-Defined API Documentationcited by 2
  32. 2022Error Identification Strategies for Python Jupyter Notebookscited by 1
  33. 2022LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognitioncited by 1
  34. 2023Dependently Typing R Vectors, Arrays, and Matricescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  2. 2026DARE: Aligning LLM Agents with the R Statistical Ecosystem via Distribution-Aware Retrievalcited by 1
  3. 2026CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Languagecited by 0
  4. 2026A Taxonomy of Programming Languages for Code Generationcited by 0