# From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

arXiv: https://arxiv.org/abs/2605.08712 · HTML: https://mlsanity.com/paper/2605.08712/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 77 of its 111 references that are in the map.
- First posted on arXiv 9 May 2026.

## Builds on (most cited first, 30 of 77)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [MagicDrive: Street View Generation with Diverse 3D Geometry Control](https://mlsanity.com/paper/2310.02601/) (2023)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models](https://mlsanity.com/paper/2503.01774/) (2025)
- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model](https://mlsanity.com/paper/2404.09967/) (2024)
- [Scaling Diffusion Transformers to 16 Billion Parameters](https://mlsanity.com/paper/2407.11633/) (2024)
- [DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models](https://mlsanity.com/paper/2501.18590/) (2025)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Endora: Video Generation Models as Endoscopy Simulators](https://mlsanity.com/paper/2403.11050/) (2024)
- [ReSim: Reliable World Simulation for Autonomous Driving](https://mlsanity.com/paper/2506.09981/) (2025)
- [DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation](https://mlsanity.com/paper/2503.15208/) (2025)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)

## Built on by (most cited first, 4 of 4)

- [Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images](https://mlsanity.com/paper/2607.01633/) (2026)
- [CrossScope: A Role-Asymmetric World Model for Joint Dual-Scope Surgical Video Prediction](https://mlsanity.com/paper/2608.03211/) (2026)
- [Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation](https://mlsanity.com/paper/2610.09800/) (2026)
- [SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models](https://mlsanity.com/paper/2610.15645/) (2026)

## Abstract (arXiv)

Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery. The core difficulty is that low-dimensional control vectors must precisely govern complex image-space evolution. In this work, we propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities. Building on this representation, we introduce a hierarchically routed visual control framework that selectively activates the most relevant control modalities and motion scales. Instead of uniformly applying all control signals, our model performs hierarchical routing to dynamically allocate conditioning capacity. We further design kinematic-prior-guided routing loss functions to ensure physically meaningful, temporally stable, and efficient expert utilization. To improve efficiency, we propose a budgeted training and inference scheme that leverages routing-induced sparsity. By selectively discarding low-significance control pathways during training and execution, our approach enables adaptive computation that is complementary to standard distillation. We additionally construct a new benchmark with curated articulated annotations, obtained through human-in-the-loop semantic labeling and differentiable pose tracking, providing realistic supervision for action-conditioned surgical video generation. Extensive experiments demonstrate that our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines. Moreover, our efficient variant achieves substantial reductions in latency while maintaining strong control accuracy.