mlsanityData to 7 Oct 2026

Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

arXiv 2604.02795 ↗First posted 3 Apr 2026cs.CL cs.AIcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

33 of 43 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2018Group Normalizationcited by 1,514
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
21 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2025Process Reinforcement through Implicit Rewardscited by 478
  4. 2025Generalizing Verifiable Instruction Followingcited by 178
  5. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  6. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  7. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  8. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  9. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  10. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  11. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  12. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  13. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  14. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  15. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  16. 2025Training AI Co-Scientists Using Rubric Rewardscited by 23
  17. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  18. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 22
  19. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  20. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 9
  21. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  2. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  3. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  4. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 1
  5. 2026CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimizationcited by 0
  6. 2026ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agentscited by 0
  7. 2026Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Followingcited by 0
  8. 2026SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey on Rubric-Guided Reinforcement Learning for Language Models
  2. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
  4. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  5. 2025Reinforcement Learning with Rubric Anchors
  6. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learning
  7. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
  8. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling