mlsanityData to 7 Oct 2026

Where2Act: From Pixels to Actions for Articulated 3D Objects

arXiv 2101.02692 ↗First posted 7 Jan 2021cs.CV cs.ROcited by 179

Not on the map. Why?

Abstract (arXiv)

One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. Check the website for code and data release: https://cs.stanford.edu/~kaichun/where2act/

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

34 of 61 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  6. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  7. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  10. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  11. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  12. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
22 more
  1. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  2. 2019Implicit 3D Orientation Learning for 6D Object Detection from RGB Imagescited by 170
  3. 2019Category-Level Articulated Object Pose Estimationcited by 158
  4. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  5. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  6. 2020Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimationcited by 123
  7. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  8. 2019Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapescited by 117
  9. 2017PCPNET: Learning Local Shape Properties from Raw Point Cloudscited by 97
  10. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  11. 2020ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theorycited by 70
  12. 2018Deep Part Induction from Articulated Object Pairscited by 66
  13. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  14. 2020KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotationscited by 50
  15. 2016Learning to Push by Grasping: Using multiple tasks for effective learningcited by 36
  16. 2018The RBO Dataset of Articulated Objects and Interactionscited by 34
  17. 2020RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloudcited by 26
  18. 2018Learning Instance Segmentation by Interactioncited by 18
  19. 2020Learning About Objects by Learning to Interact with Themcited by 14
  20. 2020Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Datacited by 13
  21. 2020Predictive and Generative Neural Networks for Object Functionalitycited by 9
  22. 2018Learning Material-Aware Local Descriptors for 3D Shapescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 15; 2023: 33; 2024: 50; 2025: 29; 2026: 41.

179 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  3. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  4. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  5. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  6. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135
  7. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  8. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  9. 2023PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objectscited by 115
  10. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  11. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  12. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
38 more
  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  2. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  3. 2022AKB-48: A Real-World Articulated Object Knowledge Basecited by 81
  4. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  5. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  6. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  7. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  8. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  9. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  10. 2024Real2Code: Reconstruct Articulated Objects via Code Generationcited by 64
  11. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  12. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  13. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  14. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 54
  15. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  16. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  17. 2022Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objectscited by 49
  18. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  19. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  20. 2021O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learningcited by 46
  21. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  22. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  23. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  24. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  25. 2021Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulationcited by 44
  26. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40
  27. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
  28. 2022Understanding 3D Object Articulation in Internet Videoscited by 36
  29. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  30. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 35
  31. 2025ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splattingcited by 35
  32. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  33. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  34. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  35. 2023Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objectscited by 33
  36. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  37. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
  38. 2025EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligencecited by 32

50 most cited of 179 All 179 (JSON)