mlsanityData to 14 Oct 2026

Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

arXiv 2607.07859 ↗First posted 8 Jul 2026cs.AI cs.HC cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 35 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

22 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  8. 2020Understanding the Behaviour of Contrastive Losscited by 398
  9. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  10. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  12. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
6 more
  1. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  2. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 58
  3. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 58
  4. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  5. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  6. 2020Reinforcement learning with human advice: a surveycited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  2. 2019State Alignment-based Imitation Learning
  3. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
  4. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics
  5. 2026Language-Critique Imitation Learning from Suboptimal Demonstrations
  6. 2026Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learning
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  8. 2026Inverse RL Helps Align AI by Imitating Humans