mlsanityData to 7 Oct 2026

RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

arXiv 2402.13222 ↗First posted 20 Feb 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in a language other than English. In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem. The purpose of RoCode is to provide a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text as well as a fine-tuning set for pretrained Romanian models. Through our results and review of related works, we argue for the need to develop code models for languages other than English.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

32 of 47 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
20 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  9. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  12. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  13. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  14. 2019SPoC: Search-based Pseudocode to Codecited by 176
  15. 2018Mapping Language to Code in Programmatic Contextcited by 104
  16. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  17. 2020The birth of Romanian BERTcited by 44
  18. 2018NAPS: Natural Program Synthesis Datasetcited by 14
  19. 2020Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modelingcited by 6
  20. 2018Extracting Parallel Paragraphs from Common Crawlcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Exploring Large Language Models for Translating Romanian Computational Problems into Englishcited by 2
  2. 2024Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysiscited by 0
  3. 2025ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generationcited by 0