mlsanityData to 7 Oct 2026

Measuring The Impact Of Programming Language Distribution

arXiv 2302.01973 ↗First posted 3 Feb 2023cs.LG cs.CL cs.PLcited by 32

Not on the map. Why?

Abstract (arXiv)

Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust. To ameliorate this issue, we present the BabelCode framework for execution-based evaluation of any benchmark in any language. BabelCode enables new investigations into the qualitative performance of models' memory, runtime, and individual test case results. Additionally, we present a new code translation dataset called Translating Python Programming Puzzles (TP3) from the Python Programming Puzzles (Schuster et al. 2021) benchmark that involves translating expert-level python functions to any language. With both BabelCode and the TP3 benchmark, we investigate if balancing the distributions of 14 languages in a training dataset improves a large language model's performance on low-resource languages. Training a model on a balanced corpus results in, on average, 12.34% higher pass@k across all tasks and languages compared to the baseline. We find that this strategy achieves 66.48% better pass@k on low-resource languages at the cost of only a 12.94% decrease to high-resource languages. In our three translation tasks, this strategy yields, on average, 30.77% better low-resource pass@k while having 19.58% worse high-resource pass@k.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

35 of 44 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
23 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  3. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  4. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  6. 2020Unsupervised Translation of Programming Languagescited by 262
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  9. 2023SantaCoder: don't reach for the stars!cited by 169
  10. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  11. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  12. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  13. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  14. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  15. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  16. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  17. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  18. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  19. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  20. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  21. 2021Programming Puzzlescited by 25
  22. 2021Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generationcited by 8
  23. 2022Evaluating How Fine-tuning on Bimodal Data Effects Code Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 6; 2025: 11; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2023Scaling Data-Constrained Language Modelscited by 391
  5. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  6. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  7. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  8. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  9. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  10. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  11. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  12. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
20 more
  1. 2025Type-Constrained Code Generation with Language Modelscited by 32
  2. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  3. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 24
  4. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  5. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  6. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 19
  7. 2025IFEvalCode: Controlled Code Generationcited by 5
  8. 2025Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environmentcited by 5
  9. 2025LLMs Lean on Priors, Not Programming Language Semanticscited by 5
  10. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLMcited by 4
  11. 2025Quasar: A Programming Language Specialized for LLM Code Actionscited by 4
  12. 2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generationcited by 4
  13. 2023Source Code Data Augmentation for Deep Learning: A Surveycited by 3
  14. 2024Teaching Machines to Code: Smart Contract Translation with LLMscited by 3
  15. 2024Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Codecited by 3
  16. 2025Beyond Language Barriers: Multi-Agent Coordination for Multi-Language Code Generationcited by 1
  17. 2026DARE: Aligning LLM Agents with the R Statistical Ecosystem via Distribution-Aware Retrievalcited by 1
  18. 2024Investigating the Transferability of Code Repair for Low-Resource Programming Languagescited by 0
  19. 2025ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluationcited by 0
  20. 2025C-ing Clearly: Enhanced Binary Code Explanations using C codecited by 0