# A Survey on Online Judge Systems and Their Applications

arXiv: https://arxiv.org/abs/1710.05913 · HTML: https://mlsanity.com/paper/1710.05913/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2017.

## Built on by (most cited first, 7 of 7)

- [SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation](https://mlsanity.com/paper/2406.14991/) (2024)
- [Exploring the Robustness of Large Language Models for Solving Programming Problems](https://mlsanity.com/paper/2306.14583/) (2023)
- [Characterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites](https://mlsanity.com/paper/2102.00423/) (2021)
- [Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions](https://mlsanity.com/paper/2307.07940/) (2023)
- [MLPro: A System for Hosting Crowdsourced Machine Learning Challenges for Open-Ended Research Problems](https://mlsanity.com/paper/2204.01216/) (2022)
- [Rule-Based Error Classification for Analyzing Differences in Frequent Errors](https://mlsanity.com/paper/2311.00513/) (2023)
- [Identifying Student Profiles Within Online Judge Systems Using Explainable Artificial Intelligence](https://mlsanity.com/paper/2402.03948/) (2024)

## Abstract (arXiv)

Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.