mlsanityData to 14 Oct 2026

In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates

arXiv 2410.15612 ↗First posted 21 Oct 2024cs.LGcited by 7

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new state-action pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear local regret O(√T+log T+√Tlog T). If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret O(log T). Experiments are used to validate the proposed algorithm.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 118 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

53 of 118 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Inverse Reinforcement Learning without Reinforcement Learning26 Mar
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  3. 2022Receding Horizon Inverse Reinforcement Learning9 Jun
  4. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  5. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning31 May

Other references · 48

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021Convex optimizationcited by 2,515
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2019Meta-Learning with Implicit Gradientscited by 525
  8. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  9. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  10. 2019Deep Reinforcement Learning for Tradingcited by 73
  11. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  12. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
36 more
  1. 2020A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problemscited by 58
  2. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 58
  3. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47
  4. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40
  5. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 38
  6. 2019Competitive Gradient Descentcited by 32
  7. 2017Efficient Regret Minimization in Non-Convex Gamescited by 28
  8. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  9. 2024Dynamic and Adaptive Feature Generation with LLMcited by 27
  10. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 25
  11. 2018Learning in Non-convex Games with an Optimization Oraclecited by 24
  12. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  13. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 21
  14. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  15. 2024Real-Time Pill Identification for the Visually Impaired Using Deep Learningcited by 21
  16. 2024Optimizing Automated Picking Systems in Warehouse Robots Using Machine Learningcited by 20
  17. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  18. 2024Dynamic Fraud Detection: Integrating Reinforcement Learning into Graph Neural Networkscited by 19
  19. 2024Robust Domain Generalization for Multi-modal Object Recognitioncited by 18
  20. 2024Deep Reinforcement Learning-based Obstacle Avoidance for Robot Movement in Warehouse Environmentscited by 17
  21. 2019Adversarial Imitation Learning from Incomplete Demonstrationscited by 16
  22. 2023Efficient Gradient Approximation Method for Constrained Bilevel Optimizationcited by 15
  23. 2024AR Overlay: Training Image Pose Estimation on Curved Surface in a Synthetic Waycited by 15
  24. 2024Fostc3net:A Lightweight YOLOv5 Based On the Network Structure Optimizationcited by 12
  25. 2024TFWT: Tabular Feature Weighting with Transformercited by 12
  26. 2024Machine Learning-Based Research on the Adaptability of Adolescents to Online Educationcited by 11
  27. 2024DRAL: Deep Reinforcement Adaptive Learning for Multi-UAVs Navigation in Unknown Indoor Environmentcited by 11
  28. 2024Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Costcited by 10
  29. 2024Research on Improved U-net Based Remote Sensing Image Segmentation Algorithmcited by 10
  30. 2020Regret minimization in stochastic non-convex learning via a proximal-gradient approachcited by 9
  31. 2024Online Learning of Multiple Tasks and Their Relationships : Testing on Spam Email Data and EEG Signals Recorded in Construction Fieldscited by 8
  32. 2024Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decompositioncited by 8
  33. 2019Learning low-dimensional state embeddings and metastable clusters from time series datacited by 6
  34. 2024Optimization and Application of Cloud-based Deep Learning Architecture for Multi-Source Data Predictioncited by 5
  35. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  36. 2024Learning-accelerated A* Search for Risk-aware Path Planningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Credit Risk Identification in Supply Chains Using Generative Adversarial Networkscited by 5
  2. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  3. 2025TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learningcited by 2
  4. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  5. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  6. 2026SCaLE: Switching Cost aware Learning and Explorationcited by 0
  7. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  2. 2018A Framework and Method for Online Inverse Reinforcement Learning
  3. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  4. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  5. 2021Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees
  6. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  7. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  8. 2025Inverse Reinforcement Learning via Convex Optimization