# Learning Visually Interpretable Oscillator Networks for Soft Continuum Robots from Video

arXiv: https://arxiv.org/abs/2511.18322 · HTML: https://mlsanity.com/paper/2511.18322/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 23 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition](https://mlsanity.com/paper/1710.04340/) (2017)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019)
- [Physics-informed Neural Networks to Model and Control Robots: a Theoretical and Experimental Investigation](https://mlsanity.com/paper/2305.05375/) (2023)
- [Neural Oscillators are Universal](https://mlsanity.com/paper/2305.08753/) (2023)
- [Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space](https://mlsanity.com/paper/2409.08439/) (2024)
- [Domain-decoupled Physics-informed Neural Networks with Closed-form Gradients for Fast Model Learning of Dynamical Systems](https://mlsanity.com/paper/2408.14951/) (2024)
- [Vision-Based Online Key Point Estimation of Deformable Robots](https://mlsanity.com/paper/2211.05222/) (2022)
- [Learning Low-Dimensional Strain Models of Soft Robots by Looking at the Evolution of Their Shape with Application to Model-Based Control](https://mlsanity.com/paper/2411.00138/) (2024)
- [Adaptive Model-Predictive Control of a Soft Continuum Robot Using a Physics-Informed Neural Network Based on Cosserat Rod Theory](https://mlsanity.com/paper/2508.12681/) (2025)
- [Koopman Operators for Modeling and Control of Soft Robotics](https://mlsanity.com/paper/2301.09708/) (2023)
- [Soft Synergies: Model Order Reduction of Hybrid Soft-Rigid Robots via Optimal Strain Parameterization](https://mlsanity.com/paper/2405.12959/) (2024)
- [Physics-informed Split Koopman Operators for Data-efficient Soft Robotic Simulation](https://mlsanity.com/paper/2502.00162/) (2025)

## Built on by (most cited first, 1 of 1)

- [LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting](https://mlsanity.com/paper/2608.16324/) (2026)

## Abstract (arXiv)

Learning soft continuum robot (SCR) dynamics from video offers flexibility but existing methods lack interpretability or rely on prior assumptions. Model-based approaches require prior knowledge and manual design. We bridge this gap by introducing: (1) The Attention Broadcast Decoder (ABCD), a plug-and-play module for autoencoder-based latent dynamics learning that generates pixel-accurate attention maps localizing each latent dimension's contribution while filtering static backgrounds, enabling visual interpretability via spatially grounded latents and on-image overlays. (2) Visual Oscillator Networks (VONs), a 2D latent oscillator network coupled to ABCD attention maps for on-image visualization of learned masses, coupling stiffness, and forces, thereby enabling mechanical interpretability. We validate our approach on single- and double-segment SCRs, demonstrating that ABCD-based models significantly improve multi-step prediction accuracy with 5.8x error reduction for Koopman operators and 3.5x for oscillator networks on a two-segment robot. VONs autonomously discover a chain structure of oscillators. This fully data-driven approach yields compact, mechanically interpretable models with potential relevance for future control applications.