# RatBodyFormer: Rat Body Surface from Keypoints

arXiv: https://arxiv.org/abs/2412.09599 · HTML: https://mlsanity.com/paper/2412.09599/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting](https://mlsanity.com/paper/2403.05087/) (2024)
- [DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model](https://mlsanity.com/paper/1605.03170/) (2016)
- [Towards Accurate Markerless Human Shape and Pose Estimation over Time](https://mlsanity.com/paper/1707.07548/) (2017)
- [Continuous Surface Embeddings](https://mlsanity.com/paper/2011.12438/) (2020)
- [A Survey of Non-Rigid 3D Registration](https://mlsanity.com/paper/2203.07858/) (2022)
- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images](https://mlsanity.com/paper/2007.06127/) (2020)
- [Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras](https://mlsanity.com/paper/2110.10355/) (2021)
- [Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos](https://mlsanity.com/paper/2403.17103/) (2024)
- [Transferring Dense Pose to Proximal Animal Classes](https://mlsanity.com/paper/2003.00080/) (2020)
- [PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting](https://mlsanity.com/paper/2401.12900/) (2024)
- [Unified 3D Mesh Recovery of Humans and Animals by Learning Animal Exercise](https://mlsanity.com/paper/2111.02450/) (2021)
- [Superpixels Based Marker Tracking Vs. Hue Thresholding In Rodent Biomechanics Application](https://mlsanity.com/paper/1710.06473/) (2017)

## Built on by (most cited first, 1 of 1)

- [InterPet4D: A Multimodal 4D Human-Pet Interaction Dataset for Pet Motion Generation](https://mlsanity.com/paper/2607.10287/) (2026)

## Abstract (arXiv)

Analyzing rat behavior lies at the heart of many scientific studies. Past methods for automated rodent modeling have focused on 3D pose estimation from keypoints, e.g., face and appendages. The pose, however, does not capture the rich body surface movement encoding the subtle rat behaviors like curling and stretching. The body surface lacks features that can be visually defined, evading these established keypoint-based methods. In this paper, we introduce the first method for reconstructing the rat body surface as a dense set of points by learning to predict it from the sparse keypoints that can be detected with past methods. Our method consists of two key contributions. The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points. RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning. We experimentally validate our framework with a number of real-world experiments. Our results collectively serve as a novel foundation for automated rat behavior analysis.