mlsanityData to 14 Oct 2026

Tiny Reward Models

arXiv 2507.09973 ↗First posted 14 Jul 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large decoder-based language models have become the dominant architecture for reward modeling in reinforcement learning from human feedback (RLHF). However, as reward models are increasingly deployed in test-time strategies, their inference costs become a growing concern. We present TinyRM, a family of small, bidirectional masked language models (MLMs) with as few as 400 million parameters, that rival the capabilities of models over 175 times larger on reasoning and safety preference modeling tasks. TinyRM combines FLAN-style prompting, Directional Low-Rank Adaptation (DoRA), and layer freezing to achieve strong performance on RewardBench, despite using significantly fewer resources. Our experiments suggest that small models benefit from domain-specific tuning strategies, particularly in reasoning, where lightweight finetuning methods are especially effective. While challenges remain in building generalist models and conversational preference modeling, our preliminary results highlight the promise of lightweight bidirectional architectures as efficient, scalable alternatives for preference modeling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 35 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

27 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method8 Dec
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  3. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  4. 2023Reward Collapse in Aligning Large Language Models28 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  7. 2014How transferable are features in deep neural networks?cited by 1,225
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  9. 2019Passage Re-ranking with BERTcited by 645
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  11. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  12. 2019How to Fine-Tune BERT for Text Classification?cited by 411
9 more
  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  3. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  4. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 89
  5. 2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 33
  6. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  7. 2023Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measurescited by 11
  8. 2025Towards Data-Efficient Language Models: A Child-Inspired Approach to Language Learningcited by 6
  9. 2025It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifierscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanism
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models
  3. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
  4. 2026Small Reward Models via Backward Inference
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation