mlsanityData to 7 Oct 2026

PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

arXiv 2309.15596 ↗First posted 27 Sep 2023cs.RO cs.CVcited by 56

Not on the map. Why?

Abstract (arXiv)

The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics. The dominant approaches for language-guided manipulation use 2D image representations, which face difficulties in combining multi-view cameras and inferring precise 3D positions and relationships. To address these limitations, we propose a 3D point cloud based policy called PolarNet for language-guided manipulation. It leverages carefully designed point cloud inputs, efficient point cloud encoders, and multimodal transformers to learn 3D point cloud representations and integrate them with language instructions for action prediction. PolarNet is shown to be effective and data efficient in a variety of experiments conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D approaches in both single-task and multi-task learning. It also achieves promising results on a real robot.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

48 of 54 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2020Point Transformercited by 1,462
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
36 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2022A Generalist Agentcited by 802
  4. 2020PCT: Point cloud transformercited by 730
  5. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  13. 2017Integral Human Pose Regressioncited by 255
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  15. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  16. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  17. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  18. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  19. 2017Compositional Human Pose Regressioncited by 124
  20. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  21. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  22. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  23. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  24. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  25. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  26. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  27. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  28. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  29. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  30. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  31. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  32. 2022ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Cloudscited by 54
  33. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  34. 2021SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learningcited by 16
  35. 2022End-to-End Learning to Grasp via Sampling from Object Point Cloudscited by 14
  36. 2020Learning Obstacle Representations for Neural Motion Planningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 24; 2026: 14.

56 papers.

Sort:

By area:

  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  4. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  5. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  6. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  7. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  8. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  9. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  10. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  11. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  12. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
38 more
  1. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  2. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  3. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulationcited by 24
  4. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
  5. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  6. 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 20
  7. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  8. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  9. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  10. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  11. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  12. 2025FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulationcited by 13
  13. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  14. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 10
  15. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  16. 2026SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policycited by 8
  17. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  18. 2026PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interactioncited by 6
  19. 2025Time-Unified Diffusion Policy with Action Discrimination for Robotic Manipulationcited by 5
  20. 2025Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulationcited by 5
  21. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  22. 2025ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulationcited by 4
  23. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentationcited by 3
  24. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  25. 2025Generalizable Hierarchical Skill Learning via Object-Centric Representationcited by 2
  26. 2025VERM: Leveraging Foundation Models to Create a Virtual Eye for Efficient 3D Robotic Manipulationcited by 2
  27. 2026G3VLA: Geometric inductive bias for Vision-Language-Action Modelscited by 2
  28. 2026Spatially Aware World Action Model via Geometric Latent Diffusioncited by 2
  29. 2024Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Multimodal Foundation Modelscited by 1
  30. 2025GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learningcited by 1
  31. 2025Gondola: Grounded Vision Language Planning for Robotic Manipulationcited by 1
  32. 2025Embodied Representation Alignment with Mirror Neuronscited by 1
  33. 2026Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Imagescited by 1
  34. 2026Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitationcited by 1
  35. 2026Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulationcited by 1
  36. 2026BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulationcited by 1
  37. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Modelscited by 0
  38. 2025Mini Diffuser: Fast Multi-task Diffusion Policy Training Using Two-level Mini-batchescited by 0

50 most cited of 56 All 56 (JSON)