# Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis

arXiv: https://arxiv.org/abs/2208.00081 · HTML: https://mlsanity.com/paper/2208.00081/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems](https://mlsanity.com/paper/1906.00331/) (2019)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods](https://mlsanity.com/paper/1902.08297/) (2019)
- [Adaptive Reward-Poisoning Attacks against Reinforcement Learning](https://mlsanity.com/paper/2003.12613/) (2020)
- [Policy Poisoning in Batch Reinforcement Learning and Control](https://mlsanity.com/paper/1910.05821/) (2019)
- [Meta-Q-Learning](https://mlsanity.com/paper/1910.00125/) (2019)
- [Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals](https://mlsanity.com/paper/1906.10571/) (2019)
- [Robust Policy Gradient against Strong Data Corruption](https://mlsanity.com/paper/2102.05800/) (2021)
- [The Confluence of Networks, Games and Learning](https://mlsanity.com/paper/2105.08158/) (2021)
- [Gradient methods for convex minimization: better rates under weaker conditions](https://mlsanity.com/paper/1303.4645/) (2013)
- [On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning](https://mlsanity.com/paper/2002.05135/) (2020)
- [On Convergence Rate of Adaptive Multiscale Value Function Approximation For Reinforcement Learning](https://mlsanity.com/paper/1908.08578/) (2019)
- [Blackwell Online Learning for Markov Decision Processes](https://mlsanity.com/paper/2012.14043/) (2020)
- [Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains](https://mlsanity.com/paper/2006.02579/) (2020)
- [Curriculum in Gradient-Based Meta-Reinforcement Learning](https://mlsanity.com/paper/2002.07956/) (2020)

## Built on by (most cited first, 5 of 5)

- [Scenario-Agnostic Zero-Trust Defense with Explainable Threshold Policy: A Meta-Learning Approach](https://mlsanity.com/paper/2303.03349/) (2023)
- [Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks](https://mlsanity.com/paper/2410.17431/) (2024)
- [A First Order Meta Stackelberg Method for Robust Federated Learning](https://mlsanity.com/paper/2306.13800/) (2023)
- [ADAPT: A Game-Theoretic and Neuro-Symbolic Framework for Automated Distributed Adaptive Penetration Testing](https://mlsanity.com/paper/2411.00217/) (2024)
- [Stackelberg Coupling of Online Representation Learning and Reinforcement Learning](https://mlsanity.com/paper/2508.07452/) (2025)

## Abstract (arXiv)

Meta reinforcement learning (meta RL), as a combination of meta-learning ideas and reinforcement learning (RL), enables the agent to adapt to different tasks using a few samples. However, this sampling-based adaptation also makes meta RL vulnerable to adversarial attacks. By manipulating the reward feedback from sampling processes in meta RL, an attacker can mislead the agent into building wrong knowledge from training experience, which deteriorates the agent's performance when dealing with different tasks after adaptation. This paper provides a game-theoretical underpinning for understanding this type of security risk. In particular, we formally define the sampling attack model as a Stackelberg game between the attacker and the agent, which yields a minimax formulation. It leads to two online attack schemes: Intermittent Attack and Persistent Attack, which enable the attacker to learn an optimal sampling attack, defined by an ε-first-order stationary point, within O(ε-2) iterations. These attack schemes freeride the learning progress concurrently without extra interactions with the environment. By corroborating the convergence results with numerical experiments, we observe that a minor effort of the attacker can significantly deteriorate the learning performance, and the minimax approach can also help robustify the meta RL algorithms.