# Adversarial Texture Optimization from RGB-D Scans

arXiv: https://arxiv.org/abs/2003.08400 · HTML: https://mlsanity.com/paper/2003.08400/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 30)

- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models](https://mlsanity.com/paper/2312.13913/) (2023)
- [Texturify: Generating Textures on 3D Shape Surfaces](https://mlsanity.com/paper/2204.02411/) (2022)
- [StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions](https://mlsanity.com/paper/2112.01530/) (2021)
- [3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation](https://mlsanity.com/paper/2311.09571/) (2023)
- [MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D](https://mlsanity.com/paper/2411.02336/) (2024)
- [HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars](https://mlsanity.com/paper/2112.10203/) (2021)
- [Mesh2Tex: Generating Mesh Textures from Image Queries](https://mlsanity.com/paper/2304.05868/) (2023)
- [Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation](https://mlsanity.com/paper/2205.05922/) (2022)
- [Local Implicit Ray Function for Generalizable Radiance Field Representation](https://mlsanity.com/paper/2304.12746/) (2023)
- [Deep Hybrid Self-Prior for Full 3D Mesh Generation](https://mlsanity.com/paper/2108.08017/) (2021)
- [GenN2N: Generative NeRF2NeRF Translation](https://mlsanity.com/paper/2404.02788/) (2024)
- [Progressive Learning of 3D Reconstruction Network from 2D GAN Data](https://mlsanity.com/paper/2305.11102/) (2023)
- [RHINO: Regularizing the Hash-based Implicit Neural Representation](https://mlsanity.com/paper/2309.12642/) (2023)
- [TexRO: Generating Delicate Textures of 3D Models by Recursive Optimization](https://mlsanity.com/paper/2403.15009/) (2024)
- [TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable Rendering](https://mlsanity.com/paper/2303.15060/) (2023)
- [Free-form 3D Scene Inpainting with Dual-stream GAN](https://mlsanity.com/paper/2212.08464/) (2022)
- [Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering](https://mlsanity.com/paper/2208.07003/) (2022)
- [Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera](https://mlsanity.com/paper/2108.09007/) (2021)
- [Projective Urban Texturing](https://mlsanity.com/paper/2201.10938/) (2022)
- [PointDreamer: Zero-shot 3D Textured Mesh Reconstruction from Colored Point Cloud](https://mlsanity.com/paper/2406.15811/) (2024)
- [3D PixBrush: Image-Guided Local Texture Synthesis](https://mlsanity.com/paper/2507.03731/) (2025)
- [Initialization and Alignment for Adversarial Texture Optimization](https://mlsanity.com/paper/2207.14289/) (2022)
- [Partial-View Object View Synthesis via Filtered Inversion](https://mlsanity.com/paper/2304.00673/) (2023)
- [Realistic Image Synthesis with Configurable 3D Scene Layouts](https://mlsanity.com/paper/2108.10031/) (2021)
- [GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer](https://mlsanity.com/paper/2410.00672/) (2024)
- [SeasonScapes: Learning Large-scale Re-lightable 3D Landscapes with Seasonal Variation from Sparse Webcams](https://mlsanity.com/paper/2605.09039/) (2026)
- [Advances in Neural 3D Mesh Texturing: A Survey](https://mlsanity.com/paper/2606.00137/) (2026)

## Abstract (arXiv)

Realistic color texture generation is an important step in RGB-D surface reconstruction, but remains challenging in practice due to inaccuracies in reconstructed geometry, misaligned camera poses, and view-dependent imaging artifacts. In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views. Specifically, we propose an approach to produce photorealistic textures for approximate surfaces, even from misaligned images, by learning an objective function that is robust to these errors. The key idea of our approach is to learn a patch-based conditional discriminator which guides the texture optimization to be tolerant to misalignments. Our discriminator takes a synthesized view and a real image, and evaluates whether the synthesized one is realistic, under a broadened definition of realism. We train the discriminator by providing as `real' examples pairs of input views and their misaligned versions -- so that the learned adversarial loss will tolerate errors from the scans. Experiments on synthetic and real data under quantitative or qualitative evaluation demonstrate the advantage of our approach in comparison to state of the art. Our code is publicly available with video demonstration.