mlsanityData to 14 Oct 2026

Differentiable Robot Rendering

arXiv 2410.13851 ↗First posted 17 Oct 2024cs.RO cs.CV cs.GRcited by 21

Not on the map. Why?

Abstract (arXiv)

Vision foundation models trained on massive amounts of visual data have shown unprecedented reasoning and planning skills in open-world settings. A key challenge in applying them to robotic tasks is the modality gap between visual data and action data. We introduce differentiable robot rendering, a method allowing the visual appearance of a robot body to be directly differentiable with respect to its control parameters. Our model integrates a kinematics-aware deformable model and Gaussians Splatting and is compatible with any robot form factors and degrees of freedom. We demonstrate its capability and usage in applications including reconstruction of robot poses from images and controlling robots through vision language models. Quantitative and qualitative results show that our differentiable rendering model provides effective gradients for robotic control directly from pixels, setting the foundation for the future applications of vision foundation models in robotics.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

32 of 39 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  10. 2017Neural 3D Mesh Renderercited by 437
  11. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  12. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
20 more
  1. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  2. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  3. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  4. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  5. 2019Differentiable Surface Splatting for Point-based Geometry Processingcited by 239
  6. 2017Stochastic Variational Video Predictioncited by 234
  7. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  8. 2022Masked World Models for Visual Controlcited by 170
  9. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  10. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 117
  11. 2016Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervisioncited by 106
  12. 2021Learning the Predictability of the Futurecited by 49
  13. 2019Camera-to-Robot Pose Estimation from a Single Imagecited by 48
  14. 2020Learning Graph-Convolutional Representations for Point Cloud Denoisingcited by 40
  15. 2021Single-view robot pose and joint angle estimation via render & comparecited by 33
  16. 2022Landscape Learning for Neural Network Inversioncited by 15
  17. 2022Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot Manipulationcited by 12
  18. 2023Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflectioncited by 10
  19. 2021Full-Body Visual Self-Modeling of Robot Morphologiescited by 8
  20. 2022Egocentric Visual Self-Modeling for Autonomous Robot Dynamics Prediction and Adaptationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  2. 2024RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulatorcited by 53
  3. 2024Neural Fields in Robotics: A Surveycited by 21
  4. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 19
  5. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 18
  6. 2025ScrewSplat: An End-to-End Method for Articulated Object Recognitioncited by 9
  7. 2025Accelerating Visual-Policy Learning through Parallel Differentiable Simulationcited by 5
  8. 2026Choreographing a World of Dynamic Objectscited by 5
  9. 2025ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentationcited by 3
  10. 2026NeuROK: Generative 4D Neural Object Kinematicscited by 3
  11. 2025Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisionscited by 2
  12. 2025Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planningcited by 2
9 more
  1. 2026Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videoscited by 2
  2. 2026Fiducial Exoskeletons: Image-Centric Robot State Estimationcited by 1
  3. 2026SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgerycited by 1
  4. 2025Learning High-Fidelity Robot Self-Model with Articulated 3D Gaussian Splattingcited by 0
  5. 2025An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfercited by 0
  6. 2025RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graphcited by 0
  7. 2026Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Renderingcited by 0
  8. 2026WorldString: Actionable World Representationcited by 0
  9. 2026Accelerating Visual Policy Learning with Sampling-Based Model Predictive Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering
  2. 2026Robot-Factored World Models via Robot Rendering
  3. 2024DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model
  4. 2024Real-World Robot Applications of Foundation Models: A Review
  5. 2025Foundation Model Driven Robotics: A Comprehensive Review
  6. 2026Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning
  7. 2023Multi-View Masked World Models for Visual Robotic Manipulation
  8. 2025SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding