# Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis

arXiv: https://arxiv.org/abs/2504.03877 · HTML: https://mlsanity.com/paper/2504.03877/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Recurrent Relational Networks](https://mlsanity.com/paper/1711.08028/) (2017)
- [Improving the Validity of Automatically Generated Feedback via Reinforcement Learning](https://mlsanity.com/paper/2403.01304/) (2024)
- [Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading](https://mlsanity.com/paper/2309.09338/) (2023)
- [A synthetic data approach for domain generalization of NLI models](https://mlsanity.com/paper/2402.12368/) (2024)
- [Distilling ChatGPT for Explainable Automated Student Answer Assessment](https://mlsanity.com/paper/2305.12962/) (2023)
- [Data Generation Using Large Language Models for Text Classification: An Empirical Case Study](https://mlsanity.com/paper/2407.12813/) (2024)
- [Automated Long Answer Grading with RiceChem Dataset](https://mlsanity.com/paper/2404.14316/) (2024)
- [Finetuning Transformer Models to Build ASAG System](https://mlsanity.com/paper/2109.12300/) (2021)
- [X-Shot: A Unified System to Handle Frequent, Few-shot and Zero-shot Learning Simultaneously in Classification](https://mlsanity.com/paper/2403.03863/) (2024)

## Built on by (most cited first, 1 of 1)

- [Autograder+: A Multi-Faceted AI Framework for Rich Pedagogical Feedback in Programming Education](https://mlsanity.com/paper/2510.26402/) (2025)

## Abstract (arXiv)

Formative assessment in STEM topics aims to promote student learning by identifying students' current understanding, thus targeting how to promote further learning. Previous studies suggest that the assessment performance of current generative large language models (LLMs) on constructed responses to open-ended questions is significantly lower than that of supervised classifiers trained on high-quality labeled data. However, we demonstrate that concept-based rubrics can significantly enhance LLM performance, which narrows the gap between LLMs as off-the shelf assessment tools, and smaller supervised models, which need large amounts of training data. For datasets where concept-based rubrics allow LLMs to achieve strong performance, we show that the concept-based rubrics help the same LLMs generate high quality synthetic data for training lightweight, high-performance supervised models. Our experiments span diverse STEM student response datasets with labels of varying quality, including a new real-world dataset that contains some AI-assisted responses, which introduces additional considerations.