# Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors

arXiv: https://arxiv.org/abs/2501.12633 · HTML: https://mlsanity.com/paper/2501.12633/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Neural Map: Structured Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.08360/) (2017)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Recurrent switching linear dynamical systems](https://mlsanity.com/paper/1610.08466/) (2016)
- [Imitating Language via Scalable Inverse Reinforcement Learning](https://mlsanity.com/paper/2409.01369/) (2024)
- [Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2210.01969/) (2022)
- [Multi-intention Inverse Q-learning for Interpretable Behavior Representation](https://mlsanity.com/paper/2311.13870/) (2023)
- [Inverse Reinforcement Learning with the Average Reward Criterion](https://mlsanity.com/paper/2305.14608/) (2023)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Neural mechanisms underlying the temporal organization of naturalistic animal behavior](https://mlsanity.com/paper/2203.02151/) (2022)

## Built on by (most cited first, 6 of 6)

- [Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behavior](https://mlsanity.com/paper/2506.15190/) (2025)
- [Improving Zero-Shot Offline RL via Behavioral Task Sampling](https://mlsanity.com/paper/2604.25496/) (2026)
- [Distributional Inverse Reinforcement Learning](https://mlsanity.com/paper/2510.03013/) (2025)
- [Fitting Reinforcement Learning Model to Behavioral Data under Bandits](https://mlsanity.com/paper/2511.04454/) (2025)
- [Probabilistic Recurrent Intention Switching Model](https://mlsanity.com/paper/2605.26998/) (2026)
- [Exploratory Experience Shapes the Geometry of Predictive Representations](https://mlsanity.com/paper/2605.27929/) (2026)

## Abstract (arXiv)

Traditional approaches to studying decision-making in neuroscience focus on simplified behavioral tasks where animals perform repetitive, stereotyped actions to receive explicit rewards. While informative, these methods constrain our understanding of decision-making to short timescale behaviors driven by explicit goals. In natural environments, animals exhibit more complex, long-term behaviors driven by intrinsic motivations that are often unobservable. Recent works in time-varying inverse reinforcement learning (IRL) aim to capture shifting motivations in long-term, freely moving behaviors. However, a crucial challenge remains: animals make decisions based on their history, not just their current state. To address this, we introduce SWIRL (SWitching IRL), a novel framework that extends traditional IRL by incorporating time-varying, history-dependent reward functions. SWIRL models long behavioral sequences as transitions between short-term decision-making processes, each governed by a unique reward function. SWIRL incorporates biologically plausible history dependency to capture how past decisions and environmental contexts shape behavior, offering a more accurate description of animal decision-making. We apply SWIRL to simulated and real-world animal behavior datasets and show that it outperforms models lacking history dependency, both quantitatively and qualitatively. This work presents the first IRL model to incorporate history-dependent policies and rewards to advance our understanding of complex, naturalistic decision-making in animals.