# DeepQR: Neural-based Quality Ratings for Learnersourced Multiple-Choice Questions

arXiv: https://arxiv.org/abs/2111.10058 · HTML: https://mlsanity.com/paper/2111.10058/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Results and Insights from Diagnostic Questions: The NeurIPS 2020 Education Challenge](https://mlsanity.com/paper/2104.04034/) (2021)
- [RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities](https://mlsanity.com/paper/1910.05522/) (2019)

## Built on by (most cited first, 5 of 5)

- [Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation](https://mlsanity.com/paper/2207.14000/) (2022)
- [Learnersourcing in the Age of AI: Student, Educator and Machine Partnerships for Content Creation](https://mlsanity.com/paper/2306.06386/) (2023)
- [Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?](https://mlsanity.com/paper/2407.05327/) (2024)
- [Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models](https://mlsanity.com/paper/2309.10444/) (2023)
- [Enhancing Student Performance Prediction on Learnersourced Questions with SGNN-LLM Synergy](https://mlsanity.com/paper/2309.13500/) (2023)

## Abstract (arXiv)

Automated question quality rating (AQQR) aims to evaluate question quality through computational means, thereby addressing emerging challenges in online learnersourced question repositories. Existing methods for AQQR rely solely on explicitly-defined criteria such as readability and word count, while not fully utilising the power of state-of-the-art deep-learning techniques. We propose DeepQR, a novel neural-network model for AQQR that is trained using multiple-choice-question (MCQ) datasets collected from PeerWise, a widely-used learnersourcing platform. Along with designing DeepQR, we investigate models based on explicitly-defined features, or semantic features, or both. We also introduce a self-attention mechanism to capture semantic correlations between MCQ components, and a contrastive-learning approach to acquire question representations using quality ratings. Extensive experiments on datasets collected from eight university-level courses illustrate that DeepQR has superior performance over six comparative models.