# Category-Independent Articulated Object Tracking with Factor Graphs

arXiv: https://arxiv.org/abs/2205.03721 · HTML: https://mlsanity.com/paper/2205.03721/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 7 May 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds](https://mlsanity.com/paper/2104.03437/) (2021)
- [ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory](https://mlsanity.com/paper/2008.10518/) (2020)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [UMPNet: Universal Manipulation Policy Network for Articulated Objects](https://mlsanity.com/paper/2109.05668/) (2021)
- [MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization](https://mlsanity.com/paper/2101.06605/) (2021)
- [A Probabilistic Framework for Learning Kinematic Models of Articulated Objects](https://mlsanity.com/paper/1405.7705/) (2014)
- [Visual Identification of Articulated Object Parts](https://mlsanity.com/paper/2012.00284/) (2020)
- [Multimodal Sensor Fusion with Differentiable Filters](https://mlsanity.com/paper/2010.13021/) (2020)
- [Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation](https://mlsanity.com/paper/2012.00088/) (2020)
- [Motion-based Object Segmentation based on Dense RGB-D Scene Flow](https://mlsanity.com/paper/1804.05195/) (2018)
- [Distributional Depth-Based Estimation of Object Articulation Models](https://mlsanity.com/paper/2108.05875/) (2021)
- [Learning Articulated Motions From Visual Demonstration](https://mlsanity.com/paper/1502.01659/) (2015)
- [Differentiable Factor Graph Optimization for Learning Smoothers](https://mlsanity.com/paper/2105.08257/) (2021)
- [Learning Hybrid Object Kinematics for Efficient Hierarchical Planning Under Uncertainty](https://mlsanity.com/paper/1907.09014/) (2019)
- [Kineverse: A Symbolic Articulation Model Framework for Model-Agnostic Mobile Manipulation](https://mlsanity.com/paper/2012.05362/) (2020)
- [Parts-Based Articulated Object Localization in Clutter Using Belief Propagation](https://mlsanity.com/paper/2008.02881/) (2020)
- [LEO: Learning Energy-based Models in Factor Graph Optimization](https://mlsanity.com/paper/2108.02274/) (2021)
- [Learning Tactile Models for Factor Graph-based Estimation](https://mlsanity.com/paper/2012.03768/) (2020)

## Built on by (most cited first, 24 of 24)

- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects](https://mlsanity.com/paper/2303.15782/) (2023)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [OPDMulti: Openable Part Detection for Multiple Objects](https://mlsanity.com/paper/2303.14087/) (2023)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion](https://mlsanity.com/paper/2510.25765/) (2025)
- [Articulated Object Estimation in the Wild](https://mlsanity.com/paper/2509.01708/) (2025)
- [CenterGrasp: Object-Aware Implicit Representation Learning for Simultaneous Shape Reconstruction and 6-DoF Grasp Estimation](https://mlsanity.com/paper/2312.08240/) (2023)
- [Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image](https://mlsanity.com/paper/2504.03177/) (2025)
- [LARM: A Large Articulated-Object Reconstruction Model](https://mlsanity.com/paper/2511.11563/) (2025)
- [CAPT: Category-level Articulation Estimation from a Single Point Cloud Using Transformer](https://mlsanity.com/paper/2402.17360/) (2024)
- [Articulated 3D Scene Graphs for Open-World Mobile Manipulation](https://mlsanity.com/paper/2602.16356/) (2026)
- [ScrewSplat: An End-to-End Method for Articulated Object Recognition](https://mlsanity.com/paper/2508.02146/) (2025)
- [Predict-Optimize-Distill: A Self-Improving Cycle for 4D Object Understanding](https://mlsanity.com/paper/2504.17441/) (2025)
- [Discovering Conceptual Knowledge with Analytic Ontology Templates for Articulated Objects](https://mlsanity.com/paper/2409.11702/) (2024)
- [H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions](https://mlsanity.com/paper/2210.12521/) (2022)
- [Online Estimation and Manipulation of Articulated Objects](https://mlsanity.com/paper/2601.01438/) (2026)
- [PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos](https://mlsanity.com/paper/2603.25539/) (2026)
- [CenterArt: Joint Shape Reconstruction and 6-DoF Grasp Estimation of Articulated Objects](https://mlsanity.com/paper/2404.14968/) (2024)
- [ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?](https://mlsanity.com/paper/2412.10050/) (2024)
- [Revisiting Proprioceptive Sensing for Articulated Object Manipulation](https://mlsanity.com/paper/2305.09584/) (2023)
- [Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds](https://mlsanity.com/paper/2511.05996/) (2025)
- [ArtReg: Visuo-Tactile based Pose Tracking and Manipulation of Unseen Articulated Objects](https://mlsanity.com/paper/2511.06378/) (2025)
- [ArticulateArena: A Metric for Articulated Kinematics](https://mlsanity.com/paper/2609.33931/) (2026)

## Abstract (arXiv)

Robots deployed in human-centric environments may need to manipulate a diverse range of articulated objects, such as doors, dishwashers, and cabinets. Articulated objects often come with unexpected articulation mechanisms that are inconsistent with categorical priors: for example, a drawer might rotate about a hinge joint instead of sliding open. We propose a category-independent framework for predicting the articulation models of unknown objects from sequences of RGB-D images. The prediction is performed by a two-step process: first, a visual perception module tracks object part poses from raw images, and second, a factor graph takes these poses and infers the articulation model including the current configuration between the parts as a 6D twist. We also propose a manipulation-oriented metric to evaluate predicted joint twists in terms of how well a compliant robot controller would be able to manipulate the articulated object given the predicted twist. We demonstrate that our visual perception and factor graph modules outperform baselines on simulated data and show the applicability of our factor graph on real world data.