# Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

arXiv: https://arxiv.org/abs/2001.08601 · HTML: https://mlsanity.com/paper/2001.08601/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2020.

## Built on by (most cited first, 14 of 14)

- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding](https://mlsanity.com/paper/2204.08129/) (2022)
- [Few-shot Keypoint Detection with Uncertainty Learning for Unseen Species](https://mlsanity.com/paper/2112.06183/) (2021)
- [T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information](https://mlsanity.com/paper/2104.08029/) (2021)
- [OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection](https://mlsanity.com/paper/2409.19899/) (2024)
- [Self-supervised Secondary Landmark Detection via 3D Representation Learning](https://mlsanity.com/paper/2110.00543/) (2021)
- [Towards Multi-Modal Animal Pose Estimation: A Survey and In-Depth Analysis](https://mlsanity.com/paper/2410.09312/) (2024)
- [Incremental Learning for Animal Pose Estimation using RBF k-DPP](https://mlsanity.com/paper/2110.13598/) (2021)
- [Few-Shot Unsupervised Image-to-Image Translation on complex scenes](https://mlsanity.com/paper/2106.03770/) (2021)
- [A Survey on Adversarial Image Synthesis](https://mlsanity.com/paper/2106.16056/) (2021)
- [Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations](https://mlsanity.com/paper/2111.14595/) (2021)
- [Overcoming the Domain Gap in Neural Action Representations](https://mlsanity.com/paper/2112.01176/) (2021)
- [ORTexME: Occlusion-Robust Human Shape and Pose via Temporal Average Texture and Mesh Encoding](https://mlsanity.com/paper/2309.12183/) (2023)
- [Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection](https://mlsanity.com/paper/2507.07994/) (2025)

## Abstract (arXiv)

Our goal is to capture the pose of neuroscience model organisms, without using any manual supervision, to be able to study how neural circuits orchestrate behaviour. Human pose estimation attains remarkable accuracy when trained on real or simulated datasets consisting of millions of frames. However, for many applications simulated models are unrealistic and real training datasets with comprehensive annotations do not exist. We address this problem with a new sim2real domain transfer method. Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework. Our model lets us train a pose estimator on the target domain by transferring readily available body keypoint locations from the source domain to generated target images. We compare our approach with existing domain transfer methods and demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish), without requiring any manual annotation on the target domain and despite using simplistic off-the-shelf animal characters for simulation, or simple geometric shapes as models. Our new datasets, code, and trained models will be published to support future neuroscientific studies.