mlsanityData to 14 Oct 2026

Point Cloud Models Improve Visual Robustness in Robotic Learners

arXiv 2404.18926 ↗First posted 29 Apr 2024cs.RO cs.CV cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Visual control policies can encounter significant performance degradation when visual conditions like lighting or camera position differ from those seen during training -- often exhibiting sharp declines in capability even for minor differences. In this work, we examine robustness to a suite of these types of visual changes for RGB-D and point cloud based visual control policies. To perform these experiments on both model-free and model-based reinforcement learners, we introduce a novel Point Cloud World Model (PCWM) and point cloud based control policies. Our experiments show that policies that explicitly encode point clouds are significantly more robust than their RGB-D counterparts. Further, we find our proposed PCWM significantly outperforms prior works in terms of sample efficiency during training. Taken together, these results suggest reasoning about the 3D scene through point clouds can improve performance, reduce learning time, and increase robustness for robotic learners. Project Webpage: https://pvskand.github.io/projects/PCWM

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 64 of its 75 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

64 of 75 references.

References · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Layer Normalizationcited by 5,669
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2018World Modelscited by 1,513
  10. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
38 more
  1. 2020Mastering Atari with Discrete World Modelscited by 1,007
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  3. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  7. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 657
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  13. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  14. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  15. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  16. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  17. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  18. 2022Masked Visual Pre-training for Motor Controlcited by 248
  19. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  20. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  21. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  22. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  23. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  24. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  25. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  26. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  27. 2017Value Prediction Networkcited by 117
  28. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  29. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  30. 2019Observational Overfitting in Reinforcement Learningcited by 91
  31. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 86
  32. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  33. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 82
  34. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  35. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
  36. 2018Improving Regression Performance with Distributional Lossescited by 67
  37. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 66
  38. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  2. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioningcited by 14
  3. 2025PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learningcited by 10
  4. 2026AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Modelscited by 8
  5. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  6. 2025Towards Fusing Point Cloud and Visual Representations for Imitation Learningcited by 6
  7. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  8. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  9. 2025Flow with the Force Field: Learning 3D Compliant Flow Matching Policies from Force and Demonstration-Guided Simulation Datacited by 3
  10. 2026StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perceptioncited by 3
  11. 2025Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Handcited by 2
  12. 2025Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalizationcited by 1
2 more
  1. 20263D Point World Models: Point Completion Enables More Accurate Dynamics Learningcited by 1
  2. 2026VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Models
  2. 2026Fourier Features Let Agents Learn High Precision Policies with Imitation Learning
  3. 2026PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning
  4. 2025Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning
  5. 2024RISE: 3D Perception Makes Real-World Robot Imitation Simple and Effective
  6. 2026PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction
  7. 2026Capturing Visual Environment Structure Correlates with Control Performance
  8. 2022Sample Efficient Robot Learning with Structured World Models