# Unicorn: Continual Learning with a Universal, Off-policy Agent

arXiv: https://arxiv.org/abs/1802.08294 · HTML: https://mlsanity.com/paper/1802.08294/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 5 of 5)

- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Building a Subspace of Policies for Scalable Continual Learning](https://mlsanity.com/paper/2211.10445/) (2022)
- [CRIL: Continual Robot Imitation Learning via Generative and Prediction Model](https://mlsanity.com/paper/2106.09422/) (2021)
- [The Effectiveness of World Models for Continual Reinforcement Learning](https://mlsanity.com/paper/2211.15944/) (2022)
- [The Construction of Reality in an AI: A Review](https://mlsanity.com/paper/2302.05448/) (2023)

## Abstract (arXiv)

Some real-world domains are best characterized as a single task, but for others this perspective is limiting. Instead, some tasks continually grow in complexity, in tandem with the agent's competence. In continual learning, also referred to as lifelong learning, there are no explicit task boundaries or curricula. As learning agents have become more powerful, continual learning remains one of the frontiers that has resisted quick progress. To test continual learning capabilities we consider a challenging 3D domain with an implicit sequence of tasks and sparse rewards. We propose a novel agent architecture called Unicorn, which demonstrates strong continual learning and outperforms several baseline agents on the proposed domain. The agent achieves this by jointly representing and learning multiple policies efficiently, using a parallel off-policy learning setup.