mlsanityData to 7 Oct 2026

Triviality Corrected Endogenous Reward

arXiv 2604.11522 ↗First posted 13 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

25 of 35 references.

References · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2025Learning to Reason under Off-Policy Guidancecited by 266
  7. 2025Learning to Reason without External Rewardscited by 199
  8. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  9. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  10. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  11. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  12. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
13 more
  1. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  2. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 61
  3. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  4. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  5. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  6. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?cited by 23
  7. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  8. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  9. 2025Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewardscited by 11
  10. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  11. 2025Writing as a testbed for open ended agentscited by 8
  12. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 5
  13. 2025EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
  2. 2026UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
  3. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  4. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
  5. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
  6. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation
  7. 2025Confidence as a Reward: Transforming LLMs into Reward Models
  8. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning