mlsanityData to 7 Oct 2026

DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics

arXiv 2312.06408 ↗First posted 11 Dec 2023cs.LG cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver. We have developed GUI tools that enable non-expert users to specify 100 tasks inspired by real-life soft-body manipulations from online videos, which we'll make public. We leverage large language models to translate task descriptions into machine-interpretable optimization objectives. The optimization objectives can help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 72 of its 101 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

72 of 101 references.

References · 72

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
38 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  3. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  4. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2019Learning Latent Plans from Playcited by 310
  11. 2020Integrated Task and Motion Planningcited by 281
  12. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  13. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  14. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  15. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  16. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  17. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  18. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  19. 2020Learning to Control PDEs with Differentiable Physicscited by 131
  20. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  21. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  22. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  23. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  24. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
  25. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  26. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  27. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  28. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  29. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  30. 2021DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contactcited by 64
  31. 2021DiffPD: Differentiable Projective Dynamicscited by 62
  32. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 62
  33. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  34. 2021FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policycited by 54
  35. 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
  36. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  37. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  38. 2022Skill-based Model-based Reinforcement Learningcited by 50

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  2. 2024DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Modelcited by 2
  3. 2025Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planningcited by 2
  4. 2024On the Evaluation of Generative Robotic Simulationscited by 1
  5. 2026RoboWits: Unexpected Challenges for Robotic Creative Problem Solvingcited by 1
  6. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0