# iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision

arXiv: https://arxiv.org/abs/2602.12779 · HTML: https://mlsanity.com/paper/2602.12779/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2026.

## Built on by (most cited first, 3 of 3)

- [CounselReflect: Opportunities and Challenges for Designing Tools to Support Self-Reflection on Mental Health and Well-Being Conversations with AI](https://mlsanity.com/paper/2603.29429/) (2026)
- [CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging](https://mlsanity.com/paper/2605.19665/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become indispensable for evaluating writing. However, text feedback they provide is often unintelligible, generic, and not specific to user criteria. Inspired by structured rubrics in education and intelligible AI explanations, we propose iRULER following identified design guidelines to textitscaffold the review process by textitspecific criteria, providing textitjustification for score selection, and offering textitactionable revisions to target different quality levels. To textitqualify user-defined criteria, we recursively used iRULER with a rubric-of-rubrics to iteratively textitrefine rubrics. In controlled experiments on writing revision and rubric creation, iRULER most improved validated LLM-judged review scores and was perceived as most helpful and aligned compared to read-only rubric and text-based LLM feedback. Qualitative findings further support how iRULER satisfies the design guidelines for user-defined feedback. This work contributes interactive rubric tools for intelligible LLM-based review and revision of writing, and user-defined rubric creation.