# Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios

arXiv: https://arxiv.org/abs/2512.00920 · HTML: https://mlsanity.com/paper/2512.00920/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 33 of its 94 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://mlsanity.com/paper/2410.16184/) (2024)
- [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://mlsanity.com/paper/2401.06080/) (2024)
- [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024)
- [JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation](https://mlsanity.com/paper/2509.22548/) (2025)
- [How to Evaluate Reward Models for RLHF](https://mlsanity.com/paper/2410.14872/) (2024)
- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [RMB: Comprehensively Benchmarking Reward Models in LLM Alignment](https://mlsanity.com/paper/2410.09893/) (2024)
- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [M-RewardBench: Evaluating Reward Models in Multilingual Settings](https://mlsanity.com/paper/2410.15522/) (2024)
- [VisRL: Intention-Driven Visual Perception via Reinforced Reasoning](https://mlsanity.com/paper/2503.07523/) (2025)
- [Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?](https://mlsanity.com/paper/2410.05584/) (2024)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [SIFThinker: Spatially-Aware Image Focus for Visual Reasoning](https://mlsanity.com/paper/2508.06259/) (2025)
- [VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models](https://mlsanity.com/paper/2505.15801/) (2025)
- [Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning](https://mlsanity.com/paper/2505.17097/) (2025)
- [ROPO: Robust Preference Optimization for Large Language Models](https://mlsanity.com/paper/2404.04102/) (2024)
- [IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding](https://mlsanity.com/paper/2508.09456/) (2025)
- [Multi-Programming Language Sandbox for LLMs](https://mlsanity.com/paper/2410.23074/) (2024)
- [When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs](https://mlsanity.com/paper/2508.11383/) (2025)
- [Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention](https://mlsanity.com/paper/2509.22854/) (2025)

## Built on by (most cited first, 8 of 8)

- [When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems](https://mlsanity.com/paper/2606.00448/) (2026)
- [Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened](https://mlsanity.com/paper/2607.13083/) (2026)
- [The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management](https://mlsanity.com/paper/2605.23071/) (2026)
- [PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks](https://mlsanity.com/paper/2602.06663/) (2026)
- [From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defense](https://mlsanity.com/paper/2601.19448/) (2026)
- [BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models](https://mlsanity.com/paper/2605.09134/) (2026)
- [What Paired Evaluations Reveal under Visual Perturbations](https://mlsanity.com/paper/2609.35583/) (2026)
- [A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversight](https://mlsanity.com/paper/2610.00111/) (2026)

## Abstract (arXiv)

Reliable reward models (RMs) are critical for ensuring the safe alignment of large language models (LLMs). However, current RM evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering "How accurate is the RM's preference perception for given samples?", it employs scientific auditing to answer: "Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?". Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM alignment systems that are verifiably safe, more robust, and trustworthy.