# Latent Space Reinforcement Learning for Inverse Material Estimation in Food Fracture Simulation

arXiv: https://arxiv.org/abs/2606.16870 · HTML: https://mlsanity.com/paper/2606.16870/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 25 of 25)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining](https://mlsanity.com/paper/2006.09191/) (2020)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization](https://mlsanity.com/paper/2202.08450/) (2022)
- [Conservative Objective Models for Effective Offline Model-Based Optimization](https://mlsanity.com/paper/2107.06882/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Local Latent Space Bayesian Optimization over Structured Inputs](https://mlsanity.com/paper/2201.11872/) (2022)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics](https://mlsanity.com/paper/2304.14369/) (2023)
- [Omnigrasp: Grasping Diverse Objects with Simulated Humanoids](https://mlsanity.com/paper/2407.11385/) (2024)
- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [RoMA: Robust Model Adaptation for Offline Model-based Optimization](https://mlsanity.com/paper/2110.14188/) (2021)
- [RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects](https://mlsanity.com/paper/2302.11553/) (2023)
- [Latent Bayesian Optimization via Autoregressive Normalizing Flows](https://mlsanity.com/paper/2504.14889/) (2025)
- [SliceIt! -- A Dual Simulator Framework for Learning Robot Food Slicing](https://mlsanity.com/paper/2404.02569/) (2024)
- [Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics](https://mlsanity.com/paper/2308.05739/) (2023)
- [FruitNinja: 3D Object Interior Texture Generation with Gaussian Splatting](https://mlsanity.com/paper/2411.12089/) (2024)
- [CulinaryCut-VLAP: A Vision-Language-Action-Physics Framework for Food Cutting via a Force-Aware Material Point Method](https://mlsanity.com/paper/2601.06451/) (2026)

## Abstract (arXiv)

Realistic visual simulation of food manipulation requires accurate material parameters, yet these are difficult to measure directly and vary across the heterogeneous regions of a single food item. We address the inverse problem of estimating material parameters from a target description of fracture behavior in a non-differentiable continuum damage mechanics simulator. Using orange peeling as a test case, we train a neural surrogate on 2,000 forward simulations and compare Covariance Matrix Adaptation Evolution Strategy (CMA-ES, a gradient-free evolutionary optimizer) with Proximal Policy Optimization (PPO, a reinforcement learning algorithm) across the original 9-dimensional parameter space and two learned 4-dimensional latent representations. Since different oranges have different material properties, a practical inverse system must handle arbitrary targets without retraining. We train a goal-conditioned PPO policy that learns a general inverse mapping: given any target description of peeling behavior, the policy produces a material parameter estimate in a single forward pass (8 surrogate evaluations, approximately 10ms). Operating in a normalizing flow latent space with a shared surrogate evaluator, the goal-conditioned policy achieves 0.642 actual recovery when validated through the simulator, outperforming the original parameter space by 23%. A warm-start extension that initializes CMA-ES refinement from the policy's output further improves recovery to 0.828 with 540 evaluations. These findings provide a practical framework for inverse food physics and lay groundwork for vision-driven material identification from video observations of food manipulation.