# Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives

arXiv: https://arxiv.org/abs/1906.10667 · HTML: https://mlsanity.com/paper/1906.10667/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 30 of 30)

- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective](https://mlsanity.com/paper/2209.08466/) (2022)
- [Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies](https://mlsanity.com/paper/2112.05062/) (2021)
- [Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments](https://mlsanity.com/paper/2201.00042/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble](https://mlsanity.com/paper/2205.09284/) (2022)
- [Ensemble Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.02618/) (2023)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [Representation Learning in Deep RL via Discrete Information Bottleneck](https://mlsanity.com/paper/2212.13835/) (2022)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Self-Activating Neural Ensembles for Continual Reinforcement Learning](https://mlsanity.com/paper/2301.00141/) (2022)
- [Toward Robust Long Range Policy Transfer](https://mlsanity.com/paper/2103.02957/) (2021)
- [QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing](https://mlsanity.com/paper/2302.00671/) (2023)
- [Bayesian Reinforcement Learning with Limited Cognitive Load](https://mlsanity.com/paper/2305.03263/) (2023)
- [Tailored Primitive Initialization is the Secret Key to Reinforcement Learning](https://mlsanity.com/paper/2511.12429/) (2025)
- [Hierarchical Primitive Composition: Simultaneous Activation of Skills with Inconsistent Action Dimensions in Multiple Hierarchies](https://mlsanity.com/paper/2110.01833/) (2021)
- [Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces](https://mlsanity.com/paper/2507.20853/) (2025)
- [Did I do that? Blame as a means to identify controlled effects in reinforcement learning](https://mlsanity.com/paper/2106.00266/) (2021)
- [Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibility](https://mlsanity.com/paper/2502.13935/) (2025)
- [Balancing Value Underestimation and Overestimation with Realistic Actor-Critic](https://mlsanity.com/paper/2110.09712/) (2021)
- [Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization](https://mlsanity.com/paper/2212.14670/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [On the Parallels Between Evolutionary Theory and the State of AI](https://mlsanity.com/paper/2505.23774/) (2025)
- [Learning Parameterized Skills from Demonstrations](https://mlsanity.com/paper/2510.24095/) (2025)
- [Unveiling Options with Neural Decomposition](https://mlsanity.com/paper/2410.11262/) (2024)
- [Evolutionary Developmental Biology Can Serve as the Conceptual Foundation for a New Design Paradigm in Artificial Intelligence](https://mlsanity.com/paper/2506.12891/) (2025)
- [Prescribe-then-Select: Adaptive Policy Selection for Contextual Stochastic Optimization](https://mlsanity.com/paper/2509.08194/) (2025)

## Abstract (arXiv)

Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is to decompose a policy into lower-level primitives or options, and a higher-level meta-policy that triggers the appropriate behaviors for a given situation. However, the meta-policy must still produce appropriate decisions in all states. In this work, we propose a policy design that decomposes into primitives, similarly to hierarchical reinforcement learning, but without a high-level meta-policy. Instead, each primitive can decide for themselves whether they wish to act in the current state. We use an information-theoretic mechanism for enabling this decentralized decision: each primitive chooses how much information it needs about the current state to make a decision and the primitive that requests the most information about the current state acts in the world. The primitives are regularized to use as little information as possible, which leads to natural competition and specialization. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in terms of generalization.