mlsanityData to 14 Oct 2026

MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks

arXiv 2507.03162 ↗First posted 3 Jul 2025cs.CY cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has transformed various domains, particularly computer science (CS) education. These models exhibit remarkable capabilities in code-related tasks and problem-solving, raising questions about their potential and limitations in advanced CS contexts. This study presents a novel bilingual (English-Romanian) multimodal (text and image) dataset of multiple-choice questions derived from a high-level computer science competition. A particularity of our dataset is that the problems are conceived such that some of them are easier solved using reasoning on paper, while for others writing code is more efficient. We systematically evaluate State of The Art LLMs on this dataset, analyzing their performance on theoretical programming tasks. Our findings reveal the strengths and limitations of current LLMs, including the influence of language choice (English vs. Romanian), providing insights into their applicability in CS education and competition settings. We also address critical ethical considerations surrounding educational integrity and the fairness of assessments in the context of LLM usage. These discussions aim to inform future educational practices and policies. To support further research, our dataset will be made publicly available in both English and Romanian. Additionally, we release an educational application tailored for Romanian students, enabling them to self-assess using the dataset in an interactive and practice-oriented environment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

20 of 27 references.

References · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  6. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  7. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  8. 2025Competitive Programming with Large Reasoning Modelscited by 119
  9. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  10. 2023Moral Foundations of Large Language Modelscited by 85
  11. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  12. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
8 more
  1. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
  2. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  3. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  4. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19
  5. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedurecited by 13
  6. 2025FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Taskscited by 9
  7. 2025Large Language Models for Code Generation: The Practitioners Perspectivecited by 9
  8. 2022A Simple, Yet Effective Approach to Finding Biases in Code Generationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Incentives Of EdTech: A Systematic Review Of EduNLP Researchcited by 0