# Neural Texture Puppeteer: A Framework for Neural Geometry and Texture Rendering of Articulated Shapes, Enabling Re-Identification at Interactive Speed

arXiv: https://arxiv.org/abs/2311.17109 · HTML: https://mlsanity.com/paper/2311.17109/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019)
- [Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"](https://mlsanity.com/paper/1908.07201/) (2019)
- [A Discriminatively Learned CNN Embedding for Person Re-identification](https://mlsanity.com/paper/1611.05666/) (2016)
- [STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering](https://mlsanity.com/paper/2101.01602/) (2020)
- [SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images](https://mlsanity.com/paper/2010.10505/) (2020)
- [End-to-End Comparative Attention Networks for Person Re-identification](https://mlsanity.com/paper/1606.04404/) (2016)
- [Cross-View Tracking for Multi-Human 3D Pose Estimation at over 100 FPS](https://mlsanity.com/paper/2003.03972/) (2020)
- [Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos](https://mlsanity.com/paper/2004.12652/) (2020)
- [Re-Identification Supervised Texture Generation](https://mlsanity.com/paper/1904.03385/) (2019)
- [3D-POP -- An automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture](https://mlsanity.com/paper/2303.13174/) (2023)
- [3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking](https://mlsanity.com/paper/2308.15316/) (2023)

## Abstract (arXiv)

In this paper, we present a neural rendering pipeline for textured articulated shapes that we call Neural Texture Puppeteer. Our method separates geometry and texture encoding. The geometry pipeline learns to capture spatial relationships on the surface of the articulated shape from ground truth data that provides this geometric information. A texture auto-encoder makes use of this information to encode textured images into a global latent code. This global texture embedding can be efficiently trained separately from the geometry, and used in a downstream task to identify individuals. The neural texture rendering and the identification of individuals run at interactive speeds. To the best of our knowledge, we are the first to offer a promising alternative to CNN- or transformer-based approaches for re-identification of articulated individuals based on neural rendering. Realistic looking novel view and pose synthesis for different synthetic cow textures further demonstrate the quality of our method. Restricted by the availability of ground truth data for the articulated shape's geometry, the quality for real-world data synthesis is reduced. We further demonstrate the flexibility of our model for real-world data by applying a synthetic to real-world texture domain shift where we reconstruct the texture from a real-world 2D RGB image. Thus, our method can be applied to endangered species where data is limited. Our novel synthetic texture dataset NePuMoo is publicly available to inspire further development in the field of neural rendering-based re-identification.