# Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments

arXiv: https://arxiv.org/abs/2109.07855 · HTML: https://mlsanity.com/paper/2109.07855/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [AllenAct: A Framework for Embodied AI Research](https://mlsanity.com/paper/2008.12760/) (2020)
- [CHALET: Cornell House Agent Learning Environment](https://mlsanity.com/paper/1801.07357/) (2018)
- [VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning](https://mlsanity.com/paper/1903.05757/) (2019)
- [Continual Reinforcement Learning in 3D Non-stationary Environments](https://mlsanity.com/paper/1905.10112/) (2019)
- [SAILenv: Learning in Virtual Visual Environments Made Simple](https://mlsanity.com/paper/2007.08224/) (2020)

## Built on by (most cited first, 2 of 2)

- [Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams](https://mlsanity.com/paper/2204.12193/) (2022)
- [Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows](https://mlsanity.com/paper/2409.11441/) (2024)

## Abstract (arXiv)

Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. Trying to simulate this learning process in machines is a challenging task, also due to the inherent difficulty in creating conditions for designing continuously evolving dynamics that are typical of the real-world. Many existing research works usually involve training and testing of virtual agents on datasets of static images or short videos, considering sequences of distinct learning tasks. However, in order to devise continual learning algorithms that operate in more realistic conditions, it is fundamental to gain access to rich, fully customizable and controlled experimental playgrounds. Focussing on the specific case of vision, we thus propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. Scenes are composed of objects that move along variable routes with different and fully customizable timings, and randomness can also be included in their evolution. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives. These general principles are concretely implemented exploiting a recently published 3D virtual environment. The user can generate scenes without the need of having strong skills in computer graphics, since all the generation facilities are exposed through a simple high-level Python interface. We publicly share the proposed generator.