mlsanityData to 14 Oct 2026

Reinforcement Learning with Automated Auxiliary Loss Search

arXiv 2210.06041 ↗First posted 12 Oct 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learning informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automatically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size 7.5 × 1020 and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2-winner) significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability to different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

46 of 61 references.

References · 46

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2018DeepMind Control Suitecited by 956
  10. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
34 more
  1. 2017Deep Reinforcement Learning that Matterscited by 754
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  3. 2020Reinforcement Learning with Augmented Datacited by 468
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  6. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  7. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  9. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  10. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  11. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  12. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  13. 2019When to use parametric models in reinforcement learning?cited by 118
  14. 2018Evolved Policy Gradientscited by 88
  15. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  16. 2021The Shape of Learning Curves: a Reviewcited by 70
  17. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  18. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  19. 2021Evolving Reinforcement Learning Algorithmscited by 49
  20. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  21. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  22. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  23. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 32
  24. 2019Evolving Rewards to Automate Reinforcement Learningcited by 31
  25. 2021Universal Trading for Order Execution with Oracle Policy Distillationcited by 30
  26. 2018Learning to Design RNAcited by 29
  27. 2019AM-LFS: AutoML for Loss Function Searchcited by 29
  28. 2019BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameterscited by 25
  29. 2020Loss Function Search for Face Recognitioncited by 22
  30. 2021AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Taskscited by 19
  31. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  32. 2020Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networkscited by 17
  33. 2022PerfectDou: Dominating DouDizhu with Perfect Information Distillationcited by 15
  34. 2019Fast Efficient Hyperparameter Tuning for Policy Gradientscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 1; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Visual Whole-Body Control for Legged Loco-Manipulationcited by 62
  2. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  3. 2023State-wise Constrained Policy Optimizationcited by 9
  4. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  5. 2023Safe Deep Policy Adaptationcited by 9
  6. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 4
  7. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  8. 2024Explaining Relationships Among Research Paperscited by 3
  9. 2025Graph Your Own Promptcited by 2
  10. 2024Adapting Image-based RL Policies via Predicted Rewardscited by 1
  11. 2026Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Modelscited by 1
  12. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0
1 more
  1. 2026DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representationscited by 0