# Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

arXiv: https://arxiv.org/abs/2205.02834 · HTML: https://mlsanity.com/paper/2205.02834/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 80 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 30 of 44)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/1910.09165/) (2019)
- [EGO-TOPO: Environment Affordances from Egocentric Video](https://mlsanity.com/paper/2001.04583/) (2020)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Flexible Neural Representation for Physics Prediction](https://mlsanity.com/paper/1806.08047/) (2018)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [Learning Affordance Landscapes for Interaction Exploration in 3D Environments](https://mlsanity.com/paper/2008.09241/) (2020)
- [Propagation Networks for Model-Based Control Under Partial Observation](https://mlsanity.com/paper/1809.11169/) (2018)
- [Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations](https://mlsanity.com/paper/2011.03880/) (2020)
- [Visual Grounding of Learned Physical Models](https://mlsanity.com/paper/2004.13664/) (2020)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)

## Built on by (most cited first, 2 of 2)

- [Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation](https://mlsanity.com/paper/2308.10898/) (2023)
- [Functionalization via Structure Completion and Motion Rectification](https://mlsanity.com/paper/2605.18010/) (2026)

## Abstract (arXiv)

This paper studies the problem of fixing malfunctional 3D objects. While previous works focus on building passive perception models to learn the functionality from static 3D objects, we argue that functionality is reckoned with respect to the physical interactions between the object and the user. Given a malfunctional object, humans can perform mental simulations to reason about its functionality and figure out how to fix it. Inspired by this, we propose FixIt, a dataset that contains about 5k poorly-designed 3D physical objects paired with choices to fix them. To mimic humans' mental simulation process, we present FixNet, a novel framework that seamlessly incorporates perception and physical dynamics. Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix. Experimental results show that our framework outperforms baseline models by a large margin, and can generalize well to objects with similar interaction types.