# Bilinear value networks

arXiv: https://arxiv.org/abs/2204.13695 · HTML: https://mlsanity.com/paper/2204.13695/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering](https://mlsanity.com/paper/1708.01471/) (2017)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [Low-rank Bilinear Pooling for Fine-Grained Classification](https://mlsanity.com/paper/1611.05109/) (2016)
- [Planning from Pixels using Inverse Dynamics Models](https://mlsanity.com/paper/2012.02419/) (2020)

## Built on by (most cited first, 9 of 9)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2208.08133/) (2022)
- [Bounding the Optimal Value Function in Compositional Reinforcement Learning](https://mlsanity.com/paper/2303.02557/) (2023)
- [Action-Sufficient Goal Representations](https://mlsanity.com/paper/2601.22496/) (2026)
- [Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective](https://mlsanity.com/paper/2209.13046/) (2022)
- [D2 Actor Critic: Diffusion Actor Meets Distributional Critic](https://mlsanity.com/paper/2510.03508/) (2025)
- [Dual Advantage Fields](https://mlsanity.com/paper/2606.04188/) (2026)

## Abstract (arXiv)

The dominant framework for off-policy multi-goal reinforcement learning involves estimating goal conditioned Q-value function. When learning to achieve multiple goals, data efficiency is intimately connected with the generalization of the Q-function to new goals. The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks. To improve the generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f(s, a), captures the environment's local dynamics at the state s; whereas the second component, φ(s, g), captures the global relationship between the current state and the goal. We show that our bilinear decomposition scheme substantially improves data efficiency, and has superior transfer to out-of-distribution goals compared to prior methods. Empirical evidence is provided on the simulated Fetch robot task-suite and dexterous manipulation with a Shadow hand.