mlsanityData to 13 Oct 2026

Neural Neural Textures Make Sim2Real Consistent

arXiv 2206.13500 ↗First posted 27 Jun 2022cs.CV cs.GR cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and neural neural textures. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. Unlike previous algorithms, TRITON is not limited to camera movements -- it can handle the movement of objects as well, making it useful for downstream tasks such as robotic manipulation.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

36 of 42 references.

References · 36

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  4. 2022Generative Adversarial Networkscited by 5,614
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,516
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,183
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,462
  10. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,195
  11. 2016Least Squares Generative Adversarial Networkscited by 1,000
  12. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
24 more
  1. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  2. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  3. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  4. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 511
  5. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 495
  6. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 455
  7. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  8. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  9. 2020Free View Synthesiscited by 231
  10. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 212
  11. 2020Stable View Synthesiscited by 165
  12. 2021NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wildcited by 136
  13. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  14. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  15. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  16. 2021Enhancing Photorealism Enhancementcited by 74
  17. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  18. 2015Learning to Generate Images with Perceptual Similarity Metricscited by 39
  19. 2019Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translationcited by 38
  20. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  21. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  22. 2021Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learningcited by 23
  23. 2021Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Datacited by 21
  24. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 38
  2. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
  3. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26