mlsanityData to 14 Oct 2026

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

arXiv 2607.25659 ↗First posted 28 Jul 2026cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 36 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

23 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization8 Jan
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Let's Verify Step by Step31 May
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023LIMA: Less Is More for Alignmentcited by 1,100
  6. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  8. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  9. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  10. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  11. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  12. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
5 more
  1. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  2. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  3. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  4. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
  5. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learningcited by 0
  2. 2026CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
  2. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  3. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  4. 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
  5. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
  6. 2026CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
  7. 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
  8. 2026CriPO: Enhancing Rubric-based RL via Self-Distillation