# WeaveRL: Weaving Reconstruction into Scene-Aware Fabrics for Perceptive Reinforcement Learning

arXiv: https://arxiv.org/abs/2609.18685 · HTML: https://mlsanity.com/paper/2609.18685/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 22 of 22)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [Riemannian Motion Policies](https://mlsanity.com/paper/1801.02854/) (2018)
- [DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands](https://mlsanity.com/paper/2412.01791/) (2024)
- [DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics](https://mlsanity.com/paper/2407.02274/) (2024)
- [nvblox: GPU-Accelerated Incremental Signed Distance Field Mapping](https://mlsanity.com/paper/2311.00626/) (2023)
- [RMPflow: A Computational Graph for Automatic Motion Policy Generation](https://mlsanity.com/paper/1811.07049/) (2018)
- [Real-time Scalable Dense Surfel Mapping](https://mlsanity.com/paper/1909.04250/) (2019)
- [Geometric Fabrics: Generalizing Classical Mechanics to Capture the Physics of Behavior](https://mlsanity.com/paper/2109.10443/) (2021)
- [Deep Reactive Policy: Learning Reactive Manipulator Motion Planning for Dynamic Environments](https://mlsanity.com/paper/2509.06953/) (2025)
- [mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies](https://mlsanity.com/paper/2509.20297/) (2025)
- [Geometric Fabrics: a Safe Guiding Medium for Policy Learning](https://mlsanity.com/paper/2405.02250/) (2024)
- [CSSDF-Net: Safe Motion Planning Based on Neural Implicit Representations of Configuration Space Distance Field](https://mlsanity.com/paper/2603.18669/) (2026)
- [ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning](https://mlsanity.com/paper/2608.19182/) (2026)
- [Flow Motion Policy: Manipulator Motion Planning with Flow Matching Models](https://mlsanity.com/paper/2604.07084/) (2026)
- [Safe and Steerable Geometric Motion Policies for Robotic Dexterous Manipulation](https://mlsanity.com/paper/2605.21811/) (2026)

## Abstract (arXiv)

Reinforcement learning allows robots to acquire complex skills, but producing policies for geometrically complex manipulation remains difficult. A promising approach is to learn on top of collision-avoidant controllers, such as geometric fabrics. However, these approaches have relied on static, hand-specified representations of the scene. Integrating active, online 3D perception into massively parallel RL training has so far been inaccessible. We introduce a GPU-accelerated method that reconstructs the scene as a collection of surfels across thousands of parallel simulation instances during active rollouts. This lets policies operate over sensor-derived, rather than hand-specified, geometry. On a suite of collision-dense manipulation tasks, our surfel fabrics enable policies to tackle geometrically complex scenes where primitive-based baselines fail, while maintaining sim-to-real transfer. Furthermore, policies learned with a scene-aware fabric are more robust to the introduction of novel geometry at test time, improving collision-free task completion under unseen obstacles from 35% to 61%. We release our reconstruction system, training code and test dataset to spur research in this direction.