mlsanityData to 14 Oct 2026

Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios

arXiv 2512.00920 ↗First posted 30 Nov 2025cs.CLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reliable reward models (RMs) are critical for ensuring the safe alignment of large language models (LLMs). However, current RM evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering "How accurate is the RM's preference perception for given samples?", it employs scientific auditing to answer: "Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?". Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM alignment systems that are verifiably safe, more robust, and trustworthy.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 94 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

33 of 94 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models16 Nov
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future12 Apr
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  7. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings20 Oct
  8. 2024How to Evaluate Reward Models for RLHF18 Oct
  9. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  10. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?8 Oct
  11. 2024Generative Reward Models2 Oct
  12. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
3 more
  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  2. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  3. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  4. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  5. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  6. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  7. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  8. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  9. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  10. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 20
  11. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 19
  12. 2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
6 more
  1. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12
  2. 2024Multi-Programming Language Sandbox for LLMscited by 10
  3. 2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
  4. 2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
  5. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  6. 2025Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settingscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystemscited by 17
  2. 2026Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happenedcited by 7
  3. 2026The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Managementcited by 6
  4. 2026PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Taskscited by 4
  5. 2026From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defensecited by 3
  6. 2026BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Modelscited by 1
  7. 2026What Paired Evaluations Reveal under Visual Perturbationscited by 0
  8. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  2. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  3. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  4. 2025The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
  5. 2025Reward Models Identify Consistency, Not Causality
  6. 2024Interpreting Language Reward Models via Contrastive Explanations
  7. 2025Towards Reliable, Uncertainty-Aware Alignment
  8. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking