# AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

arXiv: https://arxiv.org/abs/2603.07019 · HTML: https://mlsanity.com/paper/2603.07019/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [Checklists Are Better Than Reward Models For Aligning Language Models](https://mlsanity.com/paper/2507.18624/) (2025)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation](https://mlsanity.com/paper/2410.03608/) (2024)
- [RocketEval: Efficient Automated LLM Evaluation via Grading Checklist](https://mlsanity.com/paper/2503.05142/) (2025)
- [Does My Rebuttal Matter? Insights from a Major NLP Conference](https://mlsanity.com/paper/1903.11367/) (2019)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification](https://mlsanity.com/paper/2601.15808/) (2026)
- [Learning to Judge: LLMs Designing and Applying Evaluation Rubrics](https://mlsanity.com/paper/2602.08672/) (2026)
- [The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors](https://mlsanity.com/paper/2509.04484/) (2025)
- [Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation](https://mlsanity.com/paper/2409.07355/) (2024)
- [Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?](https://mlsanity.com/paper/2508.15218/) (2025)
- [Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance](https://mlsanity.com/paper/2601.14171/) (2026)
- [Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMs](https://mlsanity.com/paper/2601.18706/) (2026)
- [RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind](https://mlsanity.com/paper/2601.15715/) (2026)
- [From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notes](https://mlsanity.com/paper/2507.17717/) (2025)

## Built on by (most cited first, 5 of 5)

- [Grounded Checklist Partial Credit for Agent Skill Trajectories](https://mlsanity.com/paper/2608.27487/) (2026)
- [Rewarding Better Thinking for LLM Preference Alignment](https://mlsanity.com/paper/2607.19824/) (2026)
- [APTER: Adaptive Post-Training with Expert-Grounded Rubrics](https://mlsanity.com/paper/2608.14212/) (2026)
- [PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators](https://mlsanity.com/paper/2609.36086/) (2026)
- [Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation](https://mlsanity.com/paper/2609.37322/) (2026)

## Abstract (arXiv)

Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator rightarrow Refiner rightarrow Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.