mlsanityData to 7 Oct 2026

Instruction-driven history-aware policies for robotic manipulations

arXiv 2209.04899 ↗First posted 11 Sep 2022cs.RO cs.AI cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 11 Sep 2022.

Builds on

62 of 89 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2016Layer Normalizationcited by 5,579
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2016OpenAI Gymcited by 1,684
  10. 2016Generative Adversarial Imitation Learningcited by 1,576
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2022A Generalist Agentcited by 802
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  9. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2018Behavioral Cloning from Observationcited by 482
  12. 2021Implicit Behavioral Cloningcited by 442
  13. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  14. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  15. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  16. 2019Learning Latent Plans from Playcited by 310
  17. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  18. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  19. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  20. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  21. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  22. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  23. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  24. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  25. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  26. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  27. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  28. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  29. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  30. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  31. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  32. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  33. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  34. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  35. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  36. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  37. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  38. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 23; 2024: 31; 2025: 25; 2026: 11.

93 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  3. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  4. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  7. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  8. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  9. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  10. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  11. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  12. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
38 more
  1. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  2. 2023Data Quality in Imitation Learningcited by 72
  3. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  4. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  5. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  6. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  7. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  8. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  10. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  11. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  12. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  13. 2023Adaptive Online Replanning with Diffusion Modelscited by 33
  14. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  15. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 32
  16. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  17. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  18. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  19. 20253D FlowMatch Actor: Unified 3D Policy for Single- and Dual-Arm Manipulationcited by 19
  20. 2023Transferring Foundation Models for Generalizable Robotic Manipulationcited by 18
  21. 2024One-Shot Imitation Learning with Invariance Matching for Robotic Manipulationcited by 18
  22. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  23. 2024Generative Image as Action Modelscited by 18
  24. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  25. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  26. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  27. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  28. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
  29. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  30. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  31. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  32. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  33. 2022Instruction-Following Agents with Multimodal Transformercited by 13
  34. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  35. 2025FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulationcited by 13
  36. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  37. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  38. 20243D-MVP: 3D Multiview Pretraining for Robotic Manipulationcited by 10

50 most cited of 93 All 93 (JSON)