# Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban

arXiv: https://arxiv.org/abs/2010.01298 · HTML: https://mlsanity.com/paper/2010.01298/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications](https://mlsanity.com/paper/2110.09461/) (2021)
- [Pushing it out of the Way: Interactive Visual Navigation](https://mlsanity.com/paper/2104.14040/) (2021)
- [R3L: Relative Representations for Reinforcement Learning](https://mlsanity.com/paper/2404.12917/) (2024)
- [Reinforcement Learning via Auxiliary Task Distillation](https://mlsanity.com/paper/2406.17168/) (2024)
- [IN-Sight: Interactive Navigation through Sight](https://mlsanity.com/paper/2408.00343/) (2024)
- [Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments](https://mlsanity.com/paper/2301.09854/) (2023)
- [Mapping representations in Reinforcement Learning via Semantic Alignment for Zero-Shot Stitching](https://mlsanity.com/paper/2503.01881/) (2025)

## Abstract (arXiv)

Intelligent robots need to achieve abstract objectives using concrete, spatiotemporally complex sensory information and motor control. Tabula rasa deep reinforcement learning (RL) has tackled demanding tasks in terms of either visual, abstract, or physical reasoning, but solving these jointly remains a formidable challenge. One recent, unsolved benchmark task that integrates these challenges is Mujoban, where a robot needs to arrange 3D warehouses generated from 2D Sokoban puzzles. We explore whether integrated tasks like Mujoban can be solved by composing RL modules together in a sense-plan-act hierarchy, where modules have well-defined roles similarly to classic robot architectures. Unlike classic architectures that are typically model-based, we use only model-free modules trained with RL or supervised learning. We find that our modular RL approach dramatically outperforms the state-of-the-art monolithic RL agent on Mujoban. Further, learned modules can be reused when, e.g., using a different robot platform to solve the same task. Together our results give strong evidence for the importance of research into modular RL designs. Project website: https://sites.google.com/view/modular-rl/