# H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions

arXiv: https://arxiv.org/abs/2210.12521 · HTML: https://mlsanity.com/paper/2210.12521/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning](https://mlsanity.com/paper/2010.13611/) (2020)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Demonstration-Guided Reinforcement Learning with Learned Skills](https://mlsanity.com/paper/2107.10253/) (2021)
- [UMPNet: Universal Manipulation Policy Network for Articulated Objects](https://mlsanity.com/paper/2109.05668/) (2021)
- [Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning](https://mlsanity.com/paper/1907.09620/) (2019)
- [AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions](https://mlsanity.com/paper/2112.00246/) (2021)
- [A Probabilistic Framework for Learning Kinematic Models of Articulated Objects](https://mlsanity.com/paper/1405.7705/) (2014)
- [DoorGym: A Scalable Door Opening Environment And Baseline Agent](https://mlsanity.com/paper/1908.01887/) (2019)
- [Category-Independent Articulated Object Tracking with Factor Graphs](https://mlsanity.com/paper/2205.03721/) (2022)
- [Probabilistic Articulated Real-Time Tracking for Robot Manipulation](https://mlsanity.com/paper/1610.04871/) (2016)
- [Learning to Play by Imitating Humans](https://mlsanity.com/paper/2006.06874/) (2020)

## Built on by (most cited first, 3 of 3)

- [Tactile-Filter: Interactive Tactile Perception for Part Mating](https://mlsanity.com/paper/2303.06034/) (2023)
- [sim2art: Accurate Articulated Object Modeling from a Single Video using Synthetic Training Data Only](https://mlsanity.com/paper/2512.07698/) (2025)
- [Articulation in Prime: Primitive-Based Articulated Object Understanding from a Single Casual Video](https://mlsanity.com/paper/2605.18645/) (2026)

## Abstract (arXiv)

The world is filled with articulated objects that are difficult to determine how to use from vision alone, e.g., a door might open inwards or outwards. Humans handle these objects with strategic trial-and-error: first pushing a door then pulling if that doesn't work. We enable these capabilities in autonomous agents by proposing "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR), a probabilistic generative framework that simultaneously generates a distribution of hypotheses about how objects articulate given input observations, captures certainty over hypotheses over time, and infer plausible actions for exploration and goal-conditioned manipulation. We compare our model with existing work in manipulating objects after a handful of exploration actions, on the PartNet-Mobility dataset. We further propose a novel PuzzleBoxes benchmark that contains locked boxes that require multiple steps to solve. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework, despite using zero training data. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.