# Efficient and robust multi-task learning in the brain with modular latent primitives

arXiv: https://arxiv.org/abs/2105.14108 · HTML: https://mlsanity.com/paper/2105.14108/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 80 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 27 of 27)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Tracking the World State with Recurrent Entity Networks](https://mlsanity.com/paper/1612.03969/) (2016)
- [Routing Networks and the Challenges of Modular and Compositional Computation](https://mlsanity.com/paper/1904.12774/) (2019)
- [Continual Learning with Adaptive Weights (CLAW)](https://mlsanity.com/paper/1911.09514/) (2019)
- [Recurrent Network Models Of Sequence Generation And Memory](https://mlsanity.com/paper/1603.04687/) (2016)
- [Continual Learning with Gated Incremental Memories for sequential data processing](https://mlsanity.com/paper/2004.04077/) (2020)

## Built on by (most cited first, 4 of 4)

- [Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks](https://mlsanity.com/paper/2405.10548/) (2024)
- [Meta-Dynamical State Space Models for Integrative Neural Data Analysis](https://mlsanity.com/paper/2410.05454/) (2024)
- [Compositional meta-learning through probabilistic task inference](https://mlsanity.com/paper/2510.01858/) (2025)
- [Inductive Biases for Zero-shot Systematic Generalization in Language-informed Reinforcement Learning](https://mlsanity.com/paper/2501.15270/) (2025)

## Abstract (arXiv)

Biological agents do not have infinite resources to learn new things. For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of new skills. In spite of that, how neural networks in the brain leverage existing knowledge to learn new computations is not well understood. In this work, we study this question in artificial recurrent neural networks (RNNs) trained on a corpus of commonly used neuroscience tasks. Combining brain-inspired inductive biases we call functional and structural, we propose a system that learns new tasks by building on top of pre-trained latent dynamics organised into separate recurrent modules. These modules, acting as prior knowledge acquired previously through evolution or development, are pre-trained on the statistics of the full corpus of tasks so as to be independent and maximally informative. The resulting model, we call a Modular Latent Primitives (MoLaP) network, allows for learning multiple tasks while keeping parameter counts, and updates, low. We also show that the skills acquired with our approach are more robust to a broad range of perturbations compared to those acquired with other multi-task learning strategies, and that generalisation to new tasks is facilitated. This work offers a new perspective on achieving efficient multi-task learning in the brain, illustrating the benefits of leveraging pre-trained latent dynamical primitives.