mlsanityData to 7 Oct 2026

Multi-Task Learning with Sequence-Conditioned Transporter Networks

arXiv 2109.07578 ↗First posted 15 Sep 2021cs.LG cs.AI cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Enabling robots to solve multiple manipulation tasks has a wide range of industrial applications. While learning-based approaches enjoy flexibility and generalizability, scaling these approaches to solve such compositional tasks remains a challenge. In this work, we aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. First, we propose a new suite of benchmark specifically aimed at compositional tasks, MultiRavens, which allows defining custom task combinations through task modules that are inspired by industrial tasks and exemplify the difficulties in vision-based learning and planning methods. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling and can efficiently learn to solve multi-task long horizon problems. Our analysis suggests that not only the new framework significantly improves pick-and-place performance on novel 10 multi-task benchmark problems, but also the multi-task learning with weighted sampling can vastly improve learning and agent performances on individual tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

36 of 46 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Group Equivariant Convolutional Networkscited by 1,154
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  10. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  12. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
24 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  3. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  4. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  5. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  6. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  7. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  8. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  9. 2020Transformers for One-Shot Visual Imitationcited by 77
  10. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  11. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  12. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  13. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  14. 2020Spatial Action Maps for Mobile Manipulationcited by 31
  15. 2020The MAGICAL Benchmark for Robust Imitationcited by 30
  16. 2020Meta Reinforcement Learning with Autonomous Inference of Subtask Dependenciescited by 25
  17. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  18. 2020Self-supervised Learning for Precise Pick-and-place without Object Modelcited by 20
  19. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  20. 2020Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Datacited by 13
  21. 2021Adversarial Environment Generation for Learning to Navigate the Webcited by 13
  22. 2017Pick and Place Without Geometric Object Modelscited by 9
  23. 2020Self-Supervised Goal-Conditioned Pick and Placecited by 5
  24. 2018Deictic Image Maps: An Abstraction For Learning Pose Invariant Manipulation Policiescited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 3; 2024: 1.

8 papers.

  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  3. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  4. 2022Transporters with Visual Foresight for Solving Unseen Rearrangement Taskscited by 9
  5. 2021Compositional Learning-based Planning for Vision POMDPscited by 8
  6. 2023Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearrangingcited by 6
  7. 2023Learning visual-based deformable object rearrangement with local graph neural networkscited by 1
  8. 2024Precise Pick-and-Place using Score-Based Diffusion Networkscited by 1