mlsanityData to 7 Oct 2026

Differentiable Visual Computing for Inverse Problems and Machine Learning

arXiv 2312.04574 ↗First posted 21 Nov 2023cs.LG cs.AI cs.GRcited by 7

Not on the map. Why?

Abstract (arXiv)

Originally designed for applications in computer graphics, visual computing (VC) methods synthesize information about physical and virtual worlds, using prescribed algorithms optimized for spatial computing. VC is used to analyze geometry, physically simulate solids, fluids, and other media, and render the world via optical techniques. These fine-tuned computations that operate explicitly on a given input solve so-called forward problems, VC excels at. By contrast, deep learning (DL) allows for the construction of general algorithmic models, side stepping the need for a purely first principles-based approach to problem solving. DL is powered by highly parameterized neural network architectures -- universal function approximators -- and gradient-based search algorithms which can efficiently search that large parameter space for optimal models. This approach is predicated by neural network differentiability, the requirement that analytic derivatives of a given problem's task metric can be computed with respect to neural network's parameters. Neural networks excel when an explicit model is not known, and neural network training solves an inverse problem in which a model is computed from data.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 57 of its 109 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

57 of 109 references.

References · 57

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  9. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  10. 2020Neural Sparse Voxel Fieldscited by 959
  11. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  12. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
38 more
  1. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  2. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  3. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  4. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  5. 2019Differentiable Convex Optimization Layerscited by 458
  6. 2017Neural 3D Mesh Renderercited by 437
  7. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  8. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2021Advances in Neural Renderingcited by 365
  11. 2021Animatable Neural Radiance Fields for Modeling Dynamic Human Bodiescited by 361
  12. 2019Deep Closest Point: Learning Representations for Point Cloud Registrationcited by 318
  13. 2017PDE-Net: Learning PDEs from Datacited by 304
  14. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  15. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  16. 2019Neural Volumes: Learning Dynamic Renderable Volumes from Imagescited by 173
  17. 2020DiffusionNet: Discretization Agnostic Learning on Surfacescited by 163
  18. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  19. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  20. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 107
  21. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  22. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  23. 2018Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flowcited by 97
  24. 2020Contact and Human Dynamics from Monocular Videocited by 94
  25. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  26. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
  27. 2021Gradients are Not All You Needcited by 82
  28. 2019A Differentiable Programming System to Bridge Machine Learning and Scientific Computingcited by 81
  29. 2020Scalable Differentiable Physics for Learning and Controlcited by 72
  30. 2019DeepLight: Learning Illumination for Unconstrained Mobile Mixed Realitycited by 63
  31. 2021DiffPD: Differentiable Projective Dynamicscited by 62
  32. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  33. 2020Neural Subdivisioncited by 49
  34. 2021NTopo: Mesh-free Topology Optimization using Implicit Neural Representationscited by 41
  35. 2017Neural networks for topology optimizationcited by 39
  36. 2022Differentiable Simulation of Soft Multi-body Systemscited by 39
  37. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  38. 2019Combining Physical Simulators and Object-Based Networks for Controlcited by 18

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 20242D Neural Fields with Learned Discontinuitiescited by 2
  2. 2024Bayesian Inverse Graphics for Few-Shot Concept Learningcited by 2
  3. 2024Solving Vision Tasks with Simple Photoreceptors Instead of Camerascited by 0
  4. 2025Bayesian Inverse Physics for Neuro-Symbolic Robot Learningcited by 0
  5. 2025A DeepONet joint Neural Tangent Kernel Hybrid Framework for Physics-Informed Inverse Source Problems and Robust Image Reconstructioncited by 0
  6. 2026CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditionscited by 0
  7. 2026Learning-guided Kansa collocation for forward and inverse PDEs beyond linearitycited by 0