# Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.03361 · HTML: https://mlsanity.com/paper/2606.03361/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation](https://mlsanity.com/paper/2601.08430/) (2026)
- [HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems](https://mlsanity.com/paper/2606.01779/) (2026)
- [RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following](https://mlsanity.com/paper/2603.25133/) (2026)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [All-Mem: Agentic Lifelong Memory via Dynamic Topology Evolution](https://mlsanity.com/paper/2603.19595/) (2026)
- [Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset](https://mlsanity.com/paper/2503.23899/) (2025)
- [A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models](https://mlsanity.com/paper/2506.18485/) (2025)
- [Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints](https://mlsanity.com/paper/2511.16139/) (2025)

## Built on by (most cited first, 4 of 4)

- [TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents](https://mlsanity.com/paper/2608.16156/) (2026)
- [RLVR2: Reinforcement Learning with Verifiable Rubric-based Ranking](https://mlsanity.com/paper/2609.23457/) (2026)
- [Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics](https://mlsanity.com/paper/2609.38847/) (2026)
- [Matched-Context Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2610.13755/) (2026)

## Abstract (arXiv)

Rubric-based rewards are increasingly used for open-ended language model post-training, but criterion-level scores are often aggregated as independent utilities. This flat scalarization ignores rubric-specified prerequisite and activation relations among criteria, allowing reward or penalty to be counted even when the condition that licenses it is absent. We call this structural reward-aggregation failure False Credit Propagation (FCP). To address this limitation, we propose ourname (Graphical Event Aggregation for Rubric rewards), a probabilistic graphical framework for dependency-aware rubric aggregation. ourname models each criterion outcome as a latent Bernoulli event in a typed rubric graph, propagates soft suppression from unsupported parent events to their children, and aggregates the resulting event probabilities into a normalized expected signed utility. This yields a linear-time reward computation that can be plugged into standard rubric-based RL pipelines without changing the outer optimization algorithm. Experiments on HealthBench, WritingBench, and PLawBench with two policy backbones show that ourname consistently improves over flat aggregation and deterministic gating, achieving relative gains of up to 15.5% over flat aggregation. FCP diagnostics further show that ourname reduces leakage by 96.5% relative to flat aggregation while preserving more licensed downstream utility than deterministic gating. Our code is publicly available at https://github.com/LvCan926/GEAR.