mlsanityData to 14 Oct 2026

Studying LLM Performance on Closed- and Open-source Data

arXiv 2402.15100 ↗First posted 23 Feb 2024cs.SE cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language models (LLMs) are finding wide use in software engineering practice. These models are extremely data-hungry, and are largely trained on open-source (OSS) code distributed with permissive licenses. In terms of actual use however, a great deal of software development still occurs in the for-profit/proprietary sphere, where the code under development is not, and never has been, in the public domain; thus, many developers, do their work, and use LLMs, in settings where the models may not be as familiar with the code under development. In such settings, do LLMs work as well as they do for OSS code? If not, what are the differences? When performance differs, what are the possible causes, and are there work-arounds? In this paper, we examine this issue using proprietary, closed-source software data from Microsoft, where most proprietary code is in C# and C++. We find that performance for C# changes little from OSS --> proprietary code, but does significantly reduce for C++; we find that this difference is attributable to differences in identifiers. We also find that some performance degradation, in some cases, can be ameliorated efficiently by in-context learning.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 35 of its 79 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

35 of 79 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  11. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  12. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
23 more
  1. 2018code2vec: Learning Distributed Representations of Codecited by 250
  2. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  3. 2017Learning to Represent Programs with Graphscited by 220
  4. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  5. 2020A Transformer-based Approach for Source Code Summarizationcited by 133
  6. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  7. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  8. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 111
  9. 2022Productivity Assessment of Neural Code Completioncited by 98
  10. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  11. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 77
  12. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  13. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  14. 2020Big Code != Big Vocabulary: Open-Vocabulary Models for Source Codecited by 54
  15. 2021On the Evaluation of Neural Code Summarizationcited by 48
  16. 2023Large Language Models and Simple, Stupid Bugscited by 45
  17. 2021Multilingual training for Software Engineeringcited by 43
  18. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  19. 2019TypeWriter: Neural Type Prediction with Search-based Validationcited by 32
  20. 2020Code to Comment "Translation": Data, Metrics, Baselining & Evaluationcited by 29
  21. 2023Better patching using LLM prompting, via Self-Consistencycited by 29
  22. 2021Memorization and Generalization in Neural Code Intelligence Modelscited by 19
  23. 2018Studying the Difference Between Natural and Programming Language Corporacited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 1.

11 papers.

  1. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  2. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  3. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  4. 2025Advanced Tool Learning and Selection System (ATLASS): A Closed-Loop Framework Using LLMcited by 8
  5. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
  6. 2025A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChatcited by 5
  7. 2025Giving AI Personalities Leads to More Human-Like Reasoningcited by 3
  8. 2025Can LLMs Generate User Stories and Assess Their Quality?cited by 2
  9. 2024The Rise and Fall(?) of Software Engineeringcited by 1
  10. 2025Large Language Models Transform Organic Synthesis From Reaction Prediction to Automationcited by 1
  11. 2026From Competition to Collaboration: Designing Sustainable Mechanisms Between LLMs and Online Forumscited by 0