mlsanityData to 14 Oct 2026

Multi-Task Reinforcement Learning Enables Parameter Scaling

arXiv 2503.05126 ↗First posted 7 Mar 2025cs.LG cs.AIcited by 8

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Multi-task reinforcement learning (MTRL) aims to endow a single agent with the ability to perform well on multiple tasks. Recent works have focused on developing novel sophisticated architectures to improve performance, often resulting in larger models; it is unclear, however, whether the performance gains are a consequence of the architecture design itself or the extra parameters. We argue that gains are mostly due to scale by demonstrating that naively scaling up a simple MTRL baseline to match parameter counts outperforms the more sophisticated architectures, and these gains benefit most from scaling the critic over the actor. Additionally, we explore the training stability advantages that come with task diversity, demonstrating that increasing the number of tasks can help mitigate plasticity loss. Our findings suggest that MTRL's simultaneous training across multiple tasks provides a natural framework for beneficial parameter scaling in reinforcement learning, challenging the need for complex architectural innovations.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 36 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

27 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  2. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  3. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data1 Mar
  4. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2021Scaling Vision Transformerscited by 999
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2019Model-Based Reinforcement Learning for Ataricited by 502
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
11 more
  1. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  3. 2023Understanding plasticity in neural networkscited by 174
  4. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  5. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  6. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  7. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  8. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  9. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 49
  10. 2022PaCo: Parameter-Compositional Multi-Task Reinforcement Learningcited by 46
  11. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 43
  2. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  3. 2025MetaVLA: Unified Meta Co-training For Efficient Embodied Adaptioncited by 5
  4. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  5. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  6. 2025Combining Reinforcement Learning and Behavior Trees for NPCs in Video Games with AMD Scholacited by 0
  7. 2025On The Presence of Double-Descent in Deep Reinforcement Learningcited by 0
  8. 2026Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
  2. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
  3. 2026Scalable Reinforcement Learning via Adaptive Batch Scaling
  4. 2026TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
  5. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  6. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Model
  7. 2026Task Specialization Fine-Tuning for Contextual Reinforcement Learning
  8. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer