mlsanityData to 7 Oct 2026

Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks

arXiv 2403.03270 ↗First posted 5 Mar 2024cs.ROcited by 16

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Visual imitation learning has achieved impressive progress in learning unimanual manipulation tasks from a small set of visual observations, thanks to the latest advances in computer vision. However, learning bimanual coordination strategies and complex object relations from bimanual visual demonstrations, as well as generalizing them to categorical objects in novel cluttered scenes remain unsolved challenges. In this paper, we extend our previous work on keypoints-based visual imitation learning (mboxK-VIL)~citegaokvil2023 to bimanual manipulation tasks. The proposed Bi-KVIL jointly extracts so-called Hybrid Master-Slave Relationships (HMSR) among objects and hands, bimanual coordination strategies, and sub-symbolic task representations. Our bimanual task representation is object-centric, embodiment-independent, and viewpoint-invariant, thus generalizing well to categorical objects in novel scenes. We evaluate our approach in various real-world applications, showcasing its ability to learn fine-grained bimanual manipulation tasks from a small number of human demonstration videos. Videos and source code are available at https://sites.google.com/view/bi-kvil.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 66 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

39 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  3. 2022Interactive Imitation Learning of Bimanual Movement Primitives28 Oct
  4. 2022K-VIL: Keypoints-based Visual Imitation Learning7 Sep
  5. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videos12 Aug
  6. 2020Deep Imitation Learning for Bimanual Robotic Manipulation11 Oct
  7. 2020A Geometric Perspective on Visual Imitation Learning5 Mar
  8. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 31

  1. 2023Segment Anythingcited by 8,158
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  6. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  7. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  8. 2021Mesh Graphormercited by 264
  9. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  10. 2023Segment and Track Anythingcited by 193
  11. 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 170
  12. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
19 more
  1. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  2. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  3. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  4. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  5. 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
  6. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  7. 2019GIFT: Learning Transformation-Invariant Dense Visual Descriptors via Group CNNscited by 47
  8. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  9. 2021kPAM 2.0: Feedback Control for Category-Level Robotic Manipulationcited by 38
  10. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  11. 2022One-Shot Transfer of Affordance Regions? AffCorrs!cited by 31
  12. 2019Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networkscited by 29
  13. 2022Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objectscited by 24
  14. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21
  15. 2022Bi-Manual Manipulation and Attachment via Sim-to-Real Reinforcement Learningcited by 13
  16. 2022NIFT: Neural Interaction Field and Template for Object Manipulationcited by 4
  17. 2020Visual Descriptor Learning from Monocular Videocited by 3
  18. 2019A Lyapunov-Based Approach to Exploit Asymmetries in Robotic Dual-Arm Task Resolutioncited by 1
  19. 2019Learning and Composing Primitive Skills for Dual-arm Manipulationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  2. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  3. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  4. 2024AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulationcited by 15
  5. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 12
  6. 2025MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondencecited by 11
  7. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  8. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Framescited by 9
  9. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 8
  10. 2024BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulationcited by 5
  11. 2026Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulationcited by 5
  12. 2024Visual Imitation Learning of Task-Oriented Object Grasping and Rearrangementcited by 3
4 more
  1. 2025Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Surveycited by 3
  2. 2025Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Predictioncited by 2
  3. 2025Leveraging Foundation Models for Multimodal Graph-Based Action Recognitioncited by 1
  4. 2026CUBic: Coordinated Unified Bimanual Perception and Control Frameworkcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation ↗
  2. 2025SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives
  3. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection
  4. 2026On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learning
  5. 2025You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations
  6. 2022A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies
  7. 2026RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation
  8. 2024InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation