# Estimating Difficulty Levels of Programming Problems with Pre-trained Model

arXiv: https://arxiv.org/abs/2406.08828 · HTML: https://mlsanity.com/paper/2406.08828/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 14 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks](https://mlsanity.com/paper/2206.03545/) (2022)

## Built on by (most cited first, 2 of 2)

- [Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks](https://mlsanity.com/paper/2511.18597/) (2025)
- [CogTax: A Four-Level Cognitive Taxonomy for Command-Line Computing Education](https://mlsanity.com/paper/2607.00140/) (2026)

## Abstract (arXiv)

As the demand for programming skills grows across industries and academia, students often turn to Programming Online Judge (POJ) platforms for coding practice and competition. The difficulty level of each programming problem serves as an essential reference for guiding students' adaptive learning. However, current methods of determining difficulty levels either require extensive expert annotations or take a long time to accumulate enough student solutions for each problem. To address this issue, we formulate the problem of automatic difficulty level estimation of each programming problem, given its textual description and a solution example of code. For tackling this problem, we propose to couple two pre-trained models, one for text modality and the other for code modality, into a unified model. We built two POJ datasets for the task and the results demonstrate the effectiveness of the proposed approach and the contributions of both modalities.