# Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting

arXiv: https://arxiv.org/abs/1810.07733 · HTML: https://mlsanity.com/paper/1810.07733/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 27 of 27)

- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2207.08485/) (2022)
- [Treating Motion as Option to Reduce Motion Dependency in Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.03138/) (2022)
- [Iteratively Selecting an Easy Reference Frame Makes Unsupervised Video Object Segmentation Easier](https://mlsanity.com/paper/2112.12402/) (2021)
- [Multi-Source Fusion and Automatic Predictor Selection for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2108.05076/) (2021)
- [Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation](https://mlsanity.com/paper/2407.03056/) (2024)
- [A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.09341/) (2022)
- [Dual Prototype Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2211.12036/) (2022)
- [Video Salient Object Detection via Contrastive Features and Attention Modules](https://mlsanity.com/paper/2111.02368/) (2021)
- [SimulFlow: Simultaneously Extracting Feature and Identifying Target for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2311.18286/) (2023)
- [Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2204.02791/) (2022)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [CoralVOS: Dataset and Benchmark for Coral Video Segmentation](https://mlsanity.com/paper/2310.01946/) (2023)
- [A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2312.05830/) (2023)
- [Adaptive Multi-source Predictor for Zero-shot Video Object Segmentation](https://mlsanity.com/paper/2303.10383/) (2023)
- [Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference](https://mlsanity.com/paper/2105.00822/) (2021)
- [Online Unsupervised Video Object Segmentation via Contrastive Motion Clustering](https://mlsanity.com/paper/2306.12048/) (2023)
- [Evaluating SAM2 for Video Semantic Segmentation](https://mlsanity.com/paper/2512.01774/) (2025)
- [Learn Fast, Segment Well: Fast Object Segmentation Learning on the iCub Robot](https://mlsanity.com/paper/2206.13462/) (2022)
- [Hierarchical Graph Pattern Understanding for Zero-Shot VOS](https://mlsanity.com/paper/2312.09525/) (2023)
- [SiamPolar: Semi-supervised Realtime Video Object Segmentation with Polar Representation](https://mlsanity.com/paper/2110.14773/) (2021)
- [MED-VT++: Unifying Multimodal Learning with a Multiscale Encoder-Decoder Video Transformer](https://mlsanity.com/paper/2304.05930/) (2023)
- [Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time](https://mlsanity.com/paper/2212.08058/) (2022)
- [Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2309.14786/) (2023)
- [Saliency-Motion Guided Trunk-Collateral Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2504.05904/) (2025)
- [Shallow Features Matter: Hierarchical Memory with Heterogeneous Interaction for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2507.22465/) (2025)
- [Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2508.14729/) (2025)

## Abstract (arXiv)

Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments, since the total number of objects and their variations can be intractable. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance "teacher" network provides pseudo-labels to adapt an appearance "student" network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Unlike previous datasets, IVOS provides manipulation tasks sequences with segmentation annotation along with the waypoints for the robot trajectories. It also provides segmentation annotation for the different transformations such as translation, scale, planar rotation, and out-of-plane rotation. Our proposed adaptation method outperforms the state-of-the-art on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.