# Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner

arXiv: https://arxiv.org/abs/2205.10802 · HTML: https://mlsanity.com/paper/2205.10802/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 5 of its 32 references that are in the map.
- First posted on arXiv 22 May 2022.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 5 of 5)

- [Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey](https://mlsanity.com/paper/2007.00753/) (2020)
- [Adversarial Radar Inference: Inverse Tracking, Identifying Cognition and Designing Smart Interference](https://mlsanity.com/paper/2008.01559/) (2020)
- [Online Learning and Profit Maximization from Revealed Preferences](https://mlsanity.com/paper/1407.7294/) (2014)
- [Meta-Cognition. An Inverse-Inverse Reinforcement Learning Approach for Cognitive Radars](https://mlsanity.com/paper/2205.01794/) (2022)
- [How can a Cognitive Radar Mask its Cognition?](https://mlsanity.com/paper/2110.08608/) (2021)

## Built on by (most cited first, 3 of 3)

- [Inverse Extended Kalman Filter -- Part II: Highly Non-Linear and Uncertain Systems](https://mlsanity.com/paper/2208.06683/) (2022)
- [How can a Radar Mask its Cognition?](https://mlsanity.com/paper/2210.11444/) (2022)
- [Inverse Unscented Kalman Filter](https://mlsanity.com/paper/2304.01698/) (2023)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) deals with estimating an agent's utility function from its actions. In this paper, we consider how an agent can hide its strategy and mitigate an adversarial IRL attack; we call this inverse IRL (I-IRL). How should the decision maker choose its response to ensure a poor reconstruction of its strategy by an adversary performing IRL to estimate the agent's strategy? This paper comprises four results: First, we present an adversarial IRL algorithm that estimates the agent's strategy while controlling the agent's utility function. Our second result for I-IRL result spoofs the IRL algorithm used by the adversary. Our I-IRL results are based on revealed preference theory in micro-economics. The key idea is for the agent to deliberately choose sub-optimal responses that sufficiently masks its true strategy. Third, we give a sample complexity result for our main I-IRL result when the agent has noisy estimates of the adversary specified utility function. Finally, we illustrate our I-IRL scheme in a radar problem where a meta-cognitive radar is trying to mitigate an adversarial target.