# Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics

arXiv: https://arxiv.org/abs/2503.23989 · HTML: https://mlsanity.com/paper/2503.23989/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs](https://mlsanity.com/paper/2311.02775/) (2023)
- [SedarEval: Automated Evaluation using Self-Adaptive Rubrics](https://mlsanity.com/paper/2501.15595/) (2025)
- [Automated Grading and Feedback Tools for Programming Education: A Systematic Review](https://mlsanity.com/paper/2306.11722/) (2023)
- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions](https://mlsanity.com/paper/2304.14993/) (2023)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation](https://mlsanity.com/paper/2406.05053/) (2024)
- [Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course?](https://mlsanity.com/paper/2312.07343/) (2023)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- [CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback](https://mlsanity.com/paper/2501.10421/) (2025)

## Built on by (most cited first, 29 of 29)

- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks](https://mlsanity.com/paper/2601.02439/) (2026)
- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)
- [CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use](https://mlsanity.com/paper/2602.12268/) (2026)
- [Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria](https://mlsanity.com/paper/2605.08354/) (2026)
- [Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics](https://mlsanity.com/paper/2512.01020/) (2025)
- [RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows](https://mlsanity.com/paper/2510.09021/) (2025)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations](https://mlsanity.com/paper/2604.27093/) (2026)
- [Learning Agent Routing From Early Experience](https://mlsanity.com/paper/2605.07180/) (2026)
- [AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following](https://mlsanity.com/paper/2606.03116/) (2026)
- [AGACCI : Affiliated Grading Agents for Criteria-Centric Interface in Educational Coding Contexts](https://mlsanity.com/paper/2507.05321/) (2025)
- [Preference-Aware Rubric Learning for Personalized Evaluation](https://mlsanity.com/paper/2605.31545/) (2026)
- [Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers](https://mlsanity.com/paper/2609.11117/) (2026)
- [neuralFOMO: Can LLMs Handle Being Second Best? Measuring Envy-Like Preferences in Multi-Agent Settings](https://mlsanity.com/paper/2512.13481/) (2025)
- [Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics](https://mlsanity.com/paper/2603.05832/) (2026)
- [CRACQ: A Multi-Dimensional Approach To Automated Document Assessment](https://mlsanity.com/paper/2510.02337/) (2025)
- [Autograder+: A Multi-Faceted AI Framework for Rich Pedagogical Feedback in Programming Education](https://mlsanity.com/paper/2510.26402/) (2025)
- [How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluation](https://mlsanity.com/paper/2512.10415/) (2025)
- [PhysicsSolutionAgent: Towards Multimodal Explanations for Numerical Physics Problem Solving](https://mlsanity.com/paper/2601.13453/) (2026)
- [Agent-Based Software Artifact Evaluation](https://mlsanity.com/paper/2602.02235/) (2026)
- [CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging](https://mlsanity.com/paper/2605.19665/) (2026)
- [Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria](https://mlsanity.com/paper/2606.03814/) (2026)
- [Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering](https://mlsanity.com/paper/2606.12422/) (2026)
- [Decomposer: Learning to Decompile Symbolic Music to Programs](https://mlsanity.com/paper/2607.01849/) (2026)
- [Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach](https://mlsanity.com/paper/2607.02432/) (2026)
- [Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation](https://mlsanity.com/paper/2608.14684/) (2026)
- [Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams](https://mlsanity.com/paper/2609.29333/) (2026)

## Abstract (arXiv)

Since the emergence of Large Language Models (LLMs) popularized by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation using LLMs has become a popular field of research, code evaluation using LLMs remains under-explored. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using question-specific rubrics tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use question-agnostic rubrics. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that question-specific rubrics significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.