# Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing

arXiv: https://arxiv.org/abs/2309.16343 · HTML: https://mlsanity.com/paper/2309.16343/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2023.

## Built on by (most cited first, 10 of 10)

- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction](https://mlsanity.com/paper/2503.07909/) (2025)
- [Articulated Object Estimation in the Wild](https://mlsanity.com/paper/2509.01708/) (2025)
- [Articulated 3D Scene Graphs for Open-World Mobile Manipulation](https://mlsanity.com/paper/2602.16356/) (2026)
- [Online Estimation and Manipulation of Articulated Objects](https://mlsanity.com/paper/2601.01438/) (2026)
- [PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos](https://mlsanity.com/paper/2603.25539/) (2026)
- [CenterArt: Joint Shape Reconstruction and 6-DoF Grasp Estimation of Articulated Objects](https://mlsanity.com/paper/2404.14968/) (2024)
- [Enabling robots to follow abstract instructions and complete complex dynamic tasks](https://mlsanity.com/paper/2406.11231/) (2024)
- [PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation](https://mlsanity.com/paper/2403.15107/) (2024)
- [ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction](https://mlsanity.com/paper/2609.21751/) (2026)

## Abstract (arXiv)

From dishwashers to cabinets, humans interact with articulated objects every day, and for a robot to assist in common manipulation tasks, it must learn a representation of articulation. Recent deep learning learning methods can provide powerful vision-based priors on the affordance of articulated objects from previous, possibly simulated, experiences. In contrast, many works estimate articulation by observing the object in motion, requiring the robot to already be interacting with the object. In this work, we propose to use the best of both worlds by introducing an online estimation method that merges vision-based affordance predictions from a neural network with interactive kinematic sensing in an analytical model. Our work has the benefit of using vision to predict an articulation model before touching the object, while also being able to update the model quickly from kinematic sensing during the interaction. In this paper, we implement a full system using shared autonomy for robotic opening of articulated objects, in particular objects in which the articulation is not apparent from vision alone. We implemented our system on a real robot and performed several autonomous closed-loop experiments in which the robot had to open a door with unknown joint while estimating the articulation online. Our system achieved an 80% success rate for autonomous opening of unknown articulated objects.