# Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.00136 · HTML: https://mlsanity.com/paper/2106.00136/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/2002.03939/) (2020)
- [Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives](https://mlsanity.com/paper/1708.09165/) (2017)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2010.02974/) (2020)
- [Provable Tensor Factorization with Missing Data](https://mlsanity.com/paper/1406.2784/) (2014)
- [Multiagent Soft Q-Learning](https://mlsanity.com/paper/1804.09817/) (2018)
- [T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor](https://mlsanity.com/paper/1904.02698/) (2019)
- [Factorized Q-Learning for Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/1809.03738/) (2018)
- [The Sample-Complexity of General Reinforcement Learning](https://mlsanity.com/paper/1308.4828/) (2013)
- [Incremental multi-domain learning with network latent tensor factorization](https://mlsanity.com/paper/1904.06345/) (2019)
- [Spectral Learning on Matrices and Tensors](https://mlsanity.com/paper/2004.07984/) (2020)
- [SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.04094/) (2019)
- [Symmetry Learning for Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1706.02999/) (2017)
- [Reinforcement Learning in Rich-Observation MDPs using Spectral Methods](https://mlsanity.com/paper/1611.03907/) (2016)

## Built on by (most cited first, 25 of 25)

- [SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07489/) (2022)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Generalization in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2202.00104/) (2022)
- [Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects](https://mlsanity.com/paper/2203.10603/) (2022)
- [Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/2201.09736/) (2022)
- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation](https://mlsanity.com/paper/2203.07322/) (2022)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models](https://mlsanity.com/paper/2406.15836/) (2024)
- [Efficient Multi-agent Reinforcement Learning by Planning](https://mlsanity.com/paper/2405.11778/) (2024)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Model based Multi-agent Reinforcement Learning with Tensor Decompositions](https://mlsanity.com/paper/2110.14524/) (2021)
- [Reinforcement Learning in Factored Action Spaces using Tensor Decompositions](https://mlsanity.com/paper/2110.14538/) (2021)
- [Effects of Spectral Normalization in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2212.05331/) (2022)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning](https://mlsanity.com/paper/2501.04879/) (2025)
- [Matrix Low-Rank Trust Region Policy Optimization](https://mlsanity.com/paper/2405.17625/) (2024)
- [Matrix Low-Rank Approximation For Policy Gradient Methods](https://mlsanity.com/paper/2405.17626/) (2024)
- [Low-Rank Tensor Decompositions for the Theory of Neural Networks](https://mlsanity.com/paper/2508.18408/) (2025)
- [Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.02180/) (2023)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Offsetting Unequal Competition through RL-assisted Incentive Schemes](https://mlsanity.com/paper/2201.01450/) (2022)
- [An Improved Multi-Agent Algorithm for Cooperative and Competitive Environments by Identifying and Encouraging Cooperation among Agents](https://mlsanity.com/paper/2508.14131/) (2025)
- [Factorized Spectral Representations for Reinforcement Learning](https://mlsanity.com/paper/2607.13498/) (2026)

## Abstract (arXiv)

Reinforcement Learning in large action spaces is a challenging problem. Cooperative multi-agent reinforcement learning (MARL) exacerbates matters by imposing various constraints on communication and observability. In this work, we consider the fundamental hurdle affecting both value-based and policy-gradient approaches: an exponential blowup of the action space with the number of agents. For value-based methods, it poses challenges in accurately representing the optimal value function. For policy gradient methods, it makes training the critic difficult and exacerbates the problem of the lagging critic. We show that from a learning theory perspective, both problems can be addressed by accurately representing the associated action-value function with a low-complexity hypothesis class. This requires accurately modelling the agent interactions in a sample efficient way. To this end, we propose a novel tensorised formulation of the Bellman equation. This gives rise to our method Tesseract, which views the Q-function as a tensor whose modes correspond to the action spaces of different agents. Algorithms derived from Tesseract decompose the Q-tensor across agents and utilise low-rank tensor approximations to model agent interactions relevant to the task. We provide PAC analysis for Tesseract-based algorithms and highlight their relevance to the class of rich observation MDPs. Empirical results in different domains confirm Tesseract's gains in sample efficiency predicted by the theory.