# Semi-Supervised Neural Processes for Articulated Object Interactions

arXiv: https://arxiv.org/abs/2412.00145 · HTML: https://mlsanity.com/paper/2412.00145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models](https://mlsanity.com/paper/2402.13181/) (2024)
- [S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency](https://mlsanity.com/paper/2009.14711/) (2020)
- [PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining](https://mlsanity.com/paper/2303.08789/) (2023)
- [Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation](https://mlsanity.com/paper/2309.04640/) (2023)
- [Visual Prediction of Priors for Articulated Object Interaction](https://mlsanity.com/paper/2006.03979/) (2020)
- [Meta-learning of semi-supervised learning from tasks with heterogeneous attribute spaces](https://mlsanity.com/paper/2311.05088/) (2023)

## Abstract (arXiv)

The scarcity of labeled action data poses a considerable challenge for developing machine learning algorithms for robotic object manipulation. It is expensive and often infeasible for a robot to interact with many objects. Conversely, visual data of objects, without interaction, is abundantly available and can be leveraged for pretraining and feature extraction. However, current methods that rely on image data for pretraining do not easily adapt to task-specific predictions, since the learned features are not guaranteed to be relevant. This paper introduces the Semi-Supervised Neural Process (SSNP): an adaptive reward-prediction model designed for scenarios in which only a small subset of objects have labeled interaction data. In addition to predicting reward labels, the latent-space of the SSNP is jointly trained with an autoencoding objective using passive data from a much larger set of objects. Jointly training with both types of data allows the model to focus more effectively on generalizable features and minimizes the need for extensive retraining, thereby reducing computational demands. The efficacy of SSNP is demonstrated through a door-opening task, leading to better performance than other semi-supervised methods, and only using a fraction of the data compared to other adaptive models.