# Influencing Towards Stable Multi-Agent Interactions

arXiv: https://arxiv.org/abs/2110.08229 · HTML: https://mlsanity.com/paper/2110.08229/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity](https://mlsanity.com/paper/1707.09183/) (2017)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations](https://mlsanity.com/paper/1308.3513/) (2013)
- [Learning to Incentivize Other Learning Agents](https://mlsanity.com/paper/2006.06051/) (2020)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.12127/) (2019)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [Multi-Agent Safe Planning with Gaussian Processes](https://mlsanity.com/paper/2008.04452/) (2020)
- [Emergent Prosociality in Multi-Agent Games Through Gifting](https://mlsanity.com/paper/2105.06593/) (2021)
- [Learning from My Partner's Actions: Roles in Decentralized Robot Teams](https://mlsanity.com/paper/1910.07613/) (2019)
- [Scaling simulation-to-real transfer by learning composable robot skills](https://mlsanity.com/paper/1809.10253/) (2018)

## Built on by (most cited first, 24 of 24)

- [Stabilize to Act: Learning to Coordinate for Bimanual Manipulation](https://mlsanity.com/paper/2309.01087/) (2023)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation](https://mlsanity.com/paper/2412.06779/) (2024)
- [Conditional Imitation Learning for Multi-Agent Games](https://mlsanity.com/paper/2201.01448/) (2022)
- [Learning Vision-based Pursuit-Evasion Robot Policies](https://mlsanity.com/paper/2308.16185/) (2023)
- [RILI: Robustly Influencing Latent Intent](https://mlsanity.com/paper/2203.12705/) (2022)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibria](https://mlsanity.com/paper/2210.16175/) (2022)
- [Learning Latent Representations to Co-Adapt to Humans](https://mlsanity.com/paper/2212.09586/) (2022)
- [iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.06236/) (2023)
- [PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems](https://mlsanity.com/paper/2308.00864/) (2023)
- [Improving Human-AI Coordination through Online Adversarial Training and Generative Models](https://mlsanity.com/paper/2504.15457/) (2025)
- [Off-Policy Evaluation for Action-Dependent Non-Stationary Environments](https://mlsanity.com/paper/2301.10330/) (2023)
- [Learning to Influence Human Behavior with Offline Reinforcement Learning](https://mlsanity.com/paper/2303.02265/) (2023)
- [A behavioural transformer for effective collaboration between a robot and a non-stationary human](https://mlsanity.com/paper/2307.13447/) (2023)
- [Multi-Agent Strategy Explanations for Human-Robot Collaboration](https://mlsanity.com/paper/2311.11955/) (2023)
- [Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game](https://mlsanity.com/paper/2402.04824/) (2024)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Modular Speaker Architecture: A Framework for Sustaining Responsibility and Contextual Integrity in Multi-Agent AI Communication](https://mlsanity.com/paper/2506.01095/) (2025)
- [Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVs](https://mlsanity.com/paper/2507.12110/) (2025)
- [Robots Influencing Humans to Reveal their Goals during Collaboration and Competition](https://mlsanity.com/paper/2609.05519/) (2026)
- [Asynchronous, Option-Based Multi-Agent Policy Gradient: A Conditional Reasoning Approach](https://mlsanity.com/paper/2203.15925/) (2022)
- [Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution](https://mlsanity.com/paper/2605.13411/) (2026)

## Abstract (arXiv)

Learning in multi-agent environments is difficult due to the non-stationarity introduced by an opponent's or partner's changing behaviors. Instead of reactively adapting to the other agent's (opponent or partner) behavior, we propose an algorithm to proactively influence the other agent's strategy to stabilize -- which can restrain the non-stationarity caused by the other agent. We learn a low-dimensional latent representation of the other agent's strategy and the dynamics of how the latent strategy evolves with respect to our robot's behavior. With this learned dynamics model, we can define an unsupervised stability reward to train our robot to deliberately influence the other agent to stabilize towards a single strategy. We demonstrate the effectiveness of stabilizing in improving efficiency of maximizing the task reward in a variety of simulated environments, including autonomous driving, emergent communication, and robotic manipulation. We show qualitative results on our website: https://sites.google.com/view/stable-marl/.