mlsanityData to 7 Oct 2026

Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning

arXiv 2410.12491 ↗First posted 16 Oct 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

26 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
10 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  4. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  5. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  6. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  7. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  8. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 37
  9. 2021Max-Margin Contrastive Learningcited by 27
  10. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  2. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RLcited by 1
  3. 2025The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectivescited by 1
  4. 2026A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Datacited by 0
  5. 2026Inverse RL Helps Align AI by Imitating Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  2. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  3. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
  7. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
  8. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment