mlsanityData to 7 Oct 2026

Simple Emergent Action Representations from Multi-Task Policy Training

arXiv 2210.09566 ↗First posted 18 Oct 2022cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such as image observations or motor torques, are inherently challenging to understand or utilize directly for downstream tasks. While sensory representations have been extensively studied, the representations of motor actions are still an area of active exploration. Our work reveals that a space containing meaningful action representations emerges when a multi-task policy network takes as inputs both states and task embeddings. Moderate constraints are added to improve its representation ability. Therefore, interpolated or composed embeddings can function as a high-level interface within this space, providing instructions to the agent for executing meaningful action sequences. Empirical results demonstrate that the proposed action representations are effective for intra-action interpolation and inter-action composition with limited or no additional learning. Furthermore, our approach exhibits superior task adaptation ability compared to strong baselines in Mujoco locomotion tasks. Our work sheds light on the promising direction of learning action representations for efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Project page: https://sites.google.com/view/emergent-action-representation/

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

42 of 57 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
30 more
  1. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2018Do Better ImageNet Models Transfer Better?cited by 560
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2016Learning to reinforcement learncited by 369
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  9. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  10. 2022Masked Visual Pre-training for Motor Controlcited by 246
  11. 2021Reinforcement Learning with Prototypical Representationscited by 179
  12. 2018Meta-Gradient Reinforcement Learningcited by 156
  13. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  14. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  15. 2021APS: Active Pretraining with Successor Featurescited by 109
  16. 2018Imitating Latent Policies from Observationcited by 101
  17. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  18. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  19. 2018Evolved Policy Gradientscited by 86
  20. 2019Learning Action Representations for Reinforcement Learningcited by 75
  21. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  22. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  23. 2016Learning to Push by Grasping: Using multiple tasks for effective learningcited by 36
  24. 2018The Sparse Manifold Transformcited by 36
  25. 2020Shared Autonomy with Learned Latent Actionscited by 34
  26. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  27. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  28. 2017Learning to Learn: Meta-Critic Networks for Sample Efficient Learningcited by 28
  29. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  30. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 2; 2026: 1.

6 papers.

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  3. 2025What Do Latent Action Models Actually Learn?cited by 30
  4. 2025Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learningcited by 11
  5. 2023DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spacescited by 0
  6. 2026MotionPyramid: Hierarchical Motion Representation and Residual Interfacescited by 0