mlsanityData to 7 Oct 2026

Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment

arXiv 2602.14844 ↗First posted 16 Feb 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

AI alignment is growing in importance, yet many current approaches learn safety behavior by directly modifying policy parameters, entangling normative constraints with the underlying policy. This often yields opaque, difficult-to-edit alignment artifacts and reduces their reuse across models or deployments, a failure mode we term Alignment Waste. We propose Interactionless Inverse Reinforcement Learning, a framework for learning inspectable, editable, and reusable reward artifacts separately from policy optimization. We further introduce the Alignment Flywheel, a human-in-the-loop lifecycle for iteratively auditing, patching, and hardening these artifacts through automated evaluation and refinement. Together, these ideas recast alignment from a disposable training expense into a durable, verifiable engineering asset.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 89 of its 169 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

89 of 169 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressions25 Sep
  2. 2025ILCL: Inverse Logic-Constraint Learning from Temporally Constrained Demonstrations15 Jul
  3. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms23 Feb
  4. 2021Offline Inverse Reinforcement Learning9 Jun
  5. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 83

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2016Progressive Neural Networkscited by 1,332
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2022Mass-Editing Memory in a Transformercited by 976
38 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  4. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  5. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  6. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  7. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  8. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2016Cooperative Inverse Reinforcement Learningcited by 298
  11. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  12. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  13. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  14. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  15. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  16. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  17. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  18. 2016Discrete Variational Autoencoderscited by 124
  19. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  20. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  21. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  22. 2020Rewriting a Deep Generative Modelcited by 105
  23. 2018Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examplescited by 93
  24. 2019ColorFool: Semantic Adversarial Colorizationcited by 89
  25. 2024What makes unlearning hard and what to do about itcited by 89
  26. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  27. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  28. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  29. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  30. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 66
  31. 2016Cooperative Training of Descriptor and Generator Networkscited by 60
  32. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  33. 2023NormBank: A Knowledge Bank of Situational Social Normscited by 60
  34. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  35. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 54
  36. 2024The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapsecited by 54
  37. 2024A Unified Framework for Model Editingcited by 54
  38. 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Alignment Flywheel: A Governance-Centric Hybrid MAS for Architecture-Agnostic Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  2. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  3. 2026Inverse RL Helps Align AI by Imitating Humans
  4. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  5. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  6. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Models
  7. 2026Explaining and Preventing Alignment Collapse in Iterative RLHF
  8. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL