# Modularity benefits reinforcement learning agents with competing homeostatic drives

arXiv: https://arxiv.org/abs/2204.06608 · HTML: https://mlsanity.com/paper/2204.06608/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 10 references that are in the map.
- First posted on arXiv 13 Apr 2022.

## Builds on (most cited first, 3 of 3)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [A Survey of Multi-Objective Sequential Decision-Making](https://mlsanity.com/paper/1402.0590/) (2014)
- [Hybrid Reward Architecture for Reinforcement Learning](https://mlsanity.com/paper/1706.04208/) (2017)

## Built on by (most cited first, 2 of 2)

- [Inferring Inference](https://mlsanity.com/paper/2310.03186/) (2023)
- [Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light](https://mlsanity.com/paper/2507.11482/) (2025)

## Abstract (arXiv)

The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.