mlsanityData to 7 Oct 2026

Multi-task real-robot data with gaze attention for dual-arm fine manipulation

arXiv 2401.07603 ↗First posted 15 Jan 2024cs.RO cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

In the field of robotic manipulation, deep imitation learning is recognized as a promising approach for acquiring manipulation skills. Additionally, learning from diverse robot datasets is considered a viable method to achieve versatility and adaptability. In such research, by learning various tasks, robots achieved generality across multiple objects. However, such multi-task robot datasets have mainly focused on single-arm tasks that are relatively imprecise, not addressing the fine-grained object manipulation that robots are expected to perform in the real world. This paper introduces a dataset of diverse object manipulations that includes dual-arm tasks and/or tasks requiring fine manipulation. To this end, we have generated dataset with 224k episodes (150 hours, 1,104 language instructions) which includes dual-arm fine tasks such as bowl-moving, pencil-case opening or banana-peeling, and this data is publicly available. Additionally, this dataset includes visual attention signals as well as dual-action labels, a signal that separates actions into a robust reaching trajectory and precise interaction with objects, and language instructions to achieve robust and precise object manipulation. We applied the dataset to our Dual-Action and Attention (DAA), a model designed for fine-grained dual arm manipulation tasks and robust against covariate shifts. The model was tested with over 7k total trials in real robot manipulation tasks, demonstrating its capability in fine manipulation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 36 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

27 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scale24 Aug
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  4. 2022Training Robots without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfer19 Feb
  5. 2022Memory-based gaze prediction in deep imitation learning for robot manipulation10 Feb
  6. 2021Transformer-based deep imitation learning for dual-arm robot manipulation1 Aug
  7. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation13 Nov

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  8. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  12. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
8 more
  1. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  2. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  3. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  4. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  5. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  6. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 54
  7. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  8. 2021Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformerscited by 8
  2. 2026Gaze-Regularized Vision-Language-Action Models for Robotic Manipulationcited by 7
  3. 2025Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulationcited by 3
  4. 2025Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottleneckscited by 2
  5. 2026SILO: Simulation-in-the-Loop Sim-to-Real Transfer for Multi-Stage Cable Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulation
  2. 2026FineART: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation
  3. 2021Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
  4. 2022Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations
  5. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  6. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation
  7. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation
  8. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration