# Test-time Adaptation vs. Training-time Generalization: A Case Study in Human Instance Segmentation using Keypoints Estimation

arXiv: https://arxiv.org/abs/2212.06242 · HTML: https://mlsanity.com/paper/2212.06242/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)
- [Pose2Seg: Detection Free Human Instance Segmentation](https://mlsanity.com/paper/1803.10683/) (2018)
- [Natural Image Matting via Guided Contextual Attention](https://mlsanity.com/paper/2001.04069/) (2020)
- [MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network](https://mlsanity.com/paper/1807.04067/) (2018)
- [Unsupervised Instance Segmentation in Microscopy Images via Panoptic Domain Adaptation and Task Re-weighting](https://mlsanity.com/paper/2005.02066/) (2020)
- [Instance-sensitive Fully Convolutional Networks](https://mlsanity.com/paper/1603.08678/) (2016)
- [Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs](https://mlsanity.com/paper/1512.06735/) (2015)
- [Fast Deep Matting for Portrait Animation on Mobile Phone](https://mlsanity.com/paper/1707.08289/) (2017)
- [What Can Help Pedestrian Detection?](https://mlsanity.com/paper/1705.02757/) (2017)
- [Domain Adaptive Nuclei Instance Segmentation and Classification via Category-aware Feature Alignment and Pseudo-labelling](https://mlsanity.com/paper/2207.01233/) (2022)
- [DARCNN: Domain Adaptive Region-based Convolutional Neural Network for Unsupervised Instance Segmentation in Biomedical Images](https://mlsanity.com/paper/2104.01325/) (2021)
- [PP-HumanSeg: Connectivity-Aware Portrait Segmentation with a Large-Scale Teleconferencing Video Dataset](https://mlsanity.com/paper/2112.07146/) (2021)
- [Pose2Instance: Harnessing Keypoints for Person Instance Segmentation](https://mlsanity.com/paper/1704.01152/) (2017)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)

## Built on by (most cited first, 5 of 5)

- [EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization](https://mlsanity.com/paper/2303.01904/) (2023)
- [TransAdapt: A Transformative Framework for Online Test Time Adaptive Semantic Segmentation](https://mlsanity.com/paper/2302.14611/) (2023)
- [Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle](https://mlsanity.com/paper/2412.01562/) (2024)
- [SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds](https://mlsanity.com/paper/2601.08982/) (2026)
- [BBoxMaskPose v2: Expanding Mutual Conditioning to 3D](https://mlsanity.com/paper/2601.15200/) (2026)

## Abstract (arXiv)

We consider the problem of improving the human instance segmentation mask quality for a given test image using keypoints estimation. We compare two alternative approaches. The first approach is a test-time adaptation (TTA) method, where we allow test-time modification of the segmentation network's weights using a single unlabeled test image. In this approach, we do not assume test-time access to the labeled source dataset. More specifically, our TTA method consists of using the keypoints estimates as pseudo labels and backpropagating them to adjust the backbone weights. The second approach is a training-time generalization (TTG) method, where we permit offline access to the labeled source dataset but not the test-time modification of weights. Furthermore, we do not assume the availability of any images from or knowledge about the target domain. Our TTG method consists of augmenting the backbone features with those generated by the keypoints head and feeding the aggregate vector to the mask head. Through a comprehensive set of ablations, we evaluate both approaches and identify several factors limiting the TTA gains. In particular, we show that in the absence of a significant domain shift, TTA may hurt and TTG show only a small gain in performance, whereas for a large domain shift, TTA gains are smaller and dependent on the heuristics used, while TTG gains are larger and robust to architectural choices.