# When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On arXiv: https://arxiv.org/abs/2603.05659 ยท HTML: https://mlsanity.com/paper/2603.05659/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. - Builds on 28 of its 51 references that are in the map. - First posted on arXiv 5 Mar 2026. ## Placed in - LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/) ## Builds on (most cited first, 28 of 28) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025) - [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025) - [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023) - [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023) - [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023) - [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025) - [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025) - [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025) - [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025) - [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025) - [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025) - [OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on](https://mlsanity.com/paper/2403.01779/) (2024) - [TryOnDiffusion: A Tale of Two UNets](https://mlsanity.com/paper/2306.08276/) (2023) - [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025) - [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025) - [CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models](https://mlsanity.com/paper/2407.15886/) (2024) - [Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback](https://mlsanity.com/paper/2510.16888/) (2025) - [GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping](https://mlsanity.com/paper/2510.22319/) (2025) - [TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models](https://mlsanity.com/paper/2512.08153/) (2025) - [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025) - [CatV2TON: Taming Diffusion Transformers for Vision-Based Virtual Try-On with Temporal Concatenation](https://mlsanity.com/paper/2501.11325/) (2025) - [OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning](https://mlsanity.com/paper/2508.21066/) (2025) - [Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All](https://mlsanity.com/paper/2401.13795/) (2024) - [SuperFlow: Training Flow Matching Models with RL on the Fly](https://mlsanity.com/paper/2512.17951/) (2025) - [CoCoEdit: Content-Consistent Image Editing via Region Regularized Reinforcement Learning](https://mlsanity.com/paper/2602.14068/) (2026) - [DiT-VTON: Diffusion Transformer Framework for Unified Multi-Category Virtual Try-On and Virtual Try-All with Integrated Image Editing](https://mlsanity.com/paper/2510.04797/) (2025) ## Built on by (most cited first, 1 of 1) - [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026) ## Abstract (arXiv) Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers. But many real-world tasks admit multiple valid outputs and lack the single ideal answer that rubric generation depends on. We identify this reference-free setting as a gap in current post-training methods and propose Implicit Error Counting (IEC) to fill it. Instead of checking what a response gets right against a rubric, IEC enumerates what it gets wrong, applying severity-weighted scores across task-relevant axes and converting them into calibrated per-aspect rewards. We show that na"ive explicit enumeration is too noisy for stable optimization, and that two design choices: implicit score emission and group calibration are necessary to make error counting a reliable reward. As a case study, we validate IEC on virtual try-on (VTO), a domain that is simultaneously too constrained for holistic scoring and too permissive for rubric-based evaluation: subtle garment errors are unacceptable, yet many output variations are correct. We introduce Cascaded Error Counting (CEC) as an evaluation metric, which tracks human preferences well (60% top-1 vs. 30% others), and curate Mismatch-DressCode (MDressBench), a benchmark with maximal attribute mismatch to stress-test reward designs. On MDressBench, IEC outperforms RaR across all metrics (CEC: 5.31 vs. 5.60 on flat references; 5.20 vs. 5.53 on non-flat). On VITON-HD and DressCode, IEC matches or surpasses six baselines on 6 of 8 perceptual metrics. These results suggest that when ideal answers are unavailable, counting errors provide a stronger signal than constructing rubrics.