mlsanityData to 7 Oct 2026

MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling

arXiv 2602.17658 ↗First posted 19 Feb 2026cs.LG cs.AI cs.ITcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward modeling is central to RLHF, RLAIF, and PPO-based alignment, but its reliability is often limited by scarce and heterogeneous human preference data. In this paper, we introduce MARS (Margin and Semantic-Aware Data Augmentation for Reward Modeling), an adaptive augmentation framework for controlled low-resource reward modeling. MARS allocates more augmentation to low-margin preference pairs and uses semantic-distance-based refinement to improve chosen-rejected contrast before generating synthetic preference samples. Across three preference datasets, two reward-model backbones, and downstream alignment evaluations, MARS improves average RewardBench performance and alignment win rates over uniform augmentation, WoN, and AdaBoost-style baselines. Ablations and independent-judge evaluations suggest that the gains are not solely explained by semantic refinement alone or GPT-4.1 judge coupling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 49 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

22 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  2. 2024BOND: Aligning LLMs with Best-of-N Distillation19 Jul
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models22 Jan
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2023Large Language Model Alignment: A Surveycited by 241
  12. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
4 more
  1. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  2. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  3. 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 45
  4. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  3. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  4. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  5. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models