mlsanityData to 7 Oct 2026

Modular Multitask Reinforcement Learning with Policy Sketches

arXiv 1611.01796 ↗First posted 6 Nov 2016cs.LG cs.NEcited by 200

Not on the map. Why?

Abstract (arXiv)

We describe a framework for multitask deep reinforcement learning guided by policy sketches. Sketches annotate tasks with sequences of named subtasks, providing information about high-level structural relationships among tasks but not how to implement them---specifically not providing the detailed guidance used by much previous work on learning policy abstractions for RL (e.g. intermediate rewards, subtask completion signals, or intrinsic motivations). To learn from sketches, we present a model that associates every subtask with a modular subpolicy, and jointly maximizes reward over full task-specific policies by tying parameters across shared subpolicies. Optimization is accomplished via a decoupled actor--critic training objective that facilitates learning common behaviors from multiple dissimilar reward functions. We evaluate the effectiveness of our approach in three environments featuring both discrete and continuous control, and with sparse rewards that can be obtained only after completing a number of high-level subgoals. Experiments show that using our approach to learn policies guided by sketches gives better performance than existing techniques for learning task-specific or shared policies, while naturally inducing a library of interpretable primitive behaviors that can be recombined to rapidly adapt to new tasks.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 42; 2022: 45; 2023: 32; 2024: 33; 2025: 27; 2026: 19.

200 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  5. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  6. 2023Training Language Models with Language Feedback at Scalecited by 125
  7. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  8. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  9. 2021Skill Induction and Planning with Latent Languagecited by 99
  10. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  11. 2023Modular Deep Learningcited by 82
  12. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
38 more
  1. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  2. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  3. 2023Learning to Model the World with Languagecited by 66
  4. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  5. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  6. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  7. 2022Training Language Models with Language Feedbackcited by 48
  8. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  9. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 44
  10. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  11. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  12. 2022PaCo: Parameter-Compositional Multi-Task Reinforcement Learningcited by 40
  13. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  14. 2023Learning adaptive planning representations with natural language guidancecited by 36
  15. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  16. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 31
  17. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  18. 2023Learning to Modulate pre-trained Models in RLcited by 28
  19. 2021Block Contextual MDPs for Continual Learningcited by 25
  20. 2021The Logical Options Frameworkcited by 24
  21. 2021Compositional Transformers for Scene Generationcited by 24
  22. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 24
  23. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  24. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  25. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  26. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  27. 2022Learning Options via Compressioncited by 22
  28. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 20
  29. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  30. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 18
  31. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  32. 2021In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specificationscited by 17
  33. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  34. 2023Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guaranteescited by 17
  35. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  36. 2021Learning Task Decomposition with Ordered Memory Policy Networkcited by 15
  37. 2022Transformers are Adaptable Task Plannerscited by 15
  38. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15

50 most cited of 200 All 200 (JSON)