mlsanityData to 7 Oct 2026

Interpretable Robotic Manipulation from Language

arXiv 2405.17047 ↗First posted 27 May 2024cs.RO cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans naturally employ linguistic instructions to convey knowledge, a process that proves significantly more complex for machines, especially within the context of multitask robotic manipulation environments. Natural language, moreover, serves as the primary medium through which humans acquire new knowledge, presenting a potentially intuitive bridge for translating concepts understandable by humans into formats that can be learned by machines. In pursuit of facilitating this integration, we introduce an explainable behavior cloning agent, named Ex-PERACT, specifically designed for manipulation tasks. This agent is distinguished by its hierarchical structure, which incorporates natural language to enhance the learning process. At the top level, the model is tasked with learning a discrete skill code, while at the bottom level, the policy network translates the problem into a voxelized grid and maps the discretized actions to voxel grids. We evaluate our method across eight challenging manipulation tasks utilizing the RLBench benchmark, demonstrating that Ex-PERACT not only achieves competitive policy performance but also effectively bridges the gap between human instructions and machine execution in complex environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

32 of 39 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  5. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  7. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
20 more
  1. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  2. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  3. 2018Self-Imitation Learningcited by 135
  4. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  5. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  6. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  7. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  8. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  9. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  10. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  11. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43
  12. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Informationcited by 41
  13. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  14. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  15. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  16. 2019Conditional Driving from Natural Language Instructionscited by 17
  17. 2021Explainable Hierarchical Imitation Learning for Robotic Drink Pouringcited by 11
  18. 2020Provable Hierarchical Imitation Learning via EMcited by 10
  19. 2020Modelling Agent Policies with Interpretable Imitation Learningcited by 6
  20. 2022Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learningcited by 2

Built on by

Not cited yet.