mlsanityData to 7 Oct 2026

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

arXiv 2211.09006 ↗First posted 16 Nov 2022cs.ROcited by 54

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 physical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://tinyurl.com/toolflownet for supplementary material.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 16 Nov 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 17; 2025: 15; 2026: 11.

54 papers.

Sort:

By area:

  1. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  2. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  3. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  6. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  7. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  8. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  9. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  10. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  11. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  12. 2023FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulationcited by 37
38 more
  1. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 27
  2. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  3. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  4. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  5. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  6. 2024Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloningcited by 19
  7. 2023Grasp Multiple Objects with One Handcited by 18
  8. 2024Composable Part-Based Manipulationcited by 17
  9. 2023GMSF: Global Matching Scene Flowcited by 14
  10. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  11. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 13
  12. 2023Robot Fleet Learning via Policy Mergingcited by 11
  13. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 10
  14. 2024MILES: Making Imitation Learning Easy with Self-Supervisioncited by 9
  15. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulationcited by 8
  16. 2025H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videoscited by 8
  17. 2026Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representationcited by 7
  18. 2023Learning Generalizable Tool-use Skills through Trajectory Generationcited by 6
  19. 2024DiffSF: Diffusion Models for Scene Flow Estimationcited by 6
  20. 2023GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learningcited by 5
  21. 2024DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learningcited by 5
  22. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  23. 2024HACMan++: Spatially-Grounded Motion Primitives for Manipulationcited by 5
  24. 2024FlowMamba: Learning Point Cloud Scene Flow with Global Motion Propagationcited by 5
  25. 2025SAS: Segment Any 3D Scene with Integrated 2D Priorscited by 5
  26. 2025TopoCut: Learning Multi-Step Cutting with Spectral Rewards and Discrete Diffusion Policiescited by 2
  27. 2026LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisitioncited by 2
  28. 2023Empir3D : A Framework for Multi-Dimensional Point Cloud Assessmentcited by 1
  29. 2024Conditional Variational Auto Encoder Based Dynamic Motion for Multi-task Imitation Learningcited by 1
  30. 2025SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policycited by 1
  31. 2025Designing Tools with Control Confidencecited by 1
  32. 2026GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentationcited by 1
  33. 2025RoboPearls: Editable Video Simulation for Robot Manipulationcited by 0
  34. 2025Deformable Cluster Manipulation via Whole-Arm Policy Learningcited by 0
  35. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  36. 2026Flow-Enabled Generalization to Human Demonstrations in Few-Shot Imitation Learningcited by 0
  37. 2026AFUN: Towards an Affordance Foundation Model for Functionality Understandingcited by 0
  38. 2026Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planningcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
  2. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulation
  3. 2025PRISM: Pointcloud Reintegrated Inference via Segmentation and Cross-attention for Manipulation
  4. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training
  5. 2025PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
  6. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
  7. 2025Towards Fusing Point Cloud and Visual Representations for Imitation Learning
  8. 2024SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy