mlsanityData to 14 Oct 2026

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

arXiv 2609.01215 ↗First posted 1 Sep 2026cs.LG cs.AI cs.ROcited by 0

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Most vision-language-action (VLA) models -- OpenVLA, π0, RT-2, RDT-1B -- are monolithic: they emit raw motor commands or short action chunks without organizing behavior into reusable abstractions, so they degrade on long-horizon tasks and resist interpretation. Existing skill-discovery methods sidestep the core question of when two action sequences are behaviorally equivalent, either clustering contrastive embeddings or delegating the judgment to a language model uncalibrated to the robot's dynamics. We introduce REFACTOR-VLA, a wake/sleep system for learning reusable skills. Its sleep phase clusters motor-program fragments under a Behavioral-Equivalence Kernel (BEK) computed from rollouts of a learned latent world model Mφ; its wake phase emits typed lambda terms over a Hindley--Milner-inspired vocabulary, consumed by a library-conditioned rectified-flow action decoder. Abstractions are admitted only if they pass Minimum Description Length and return-preservation gates. On LIBERO we report two findings. First, enlarging the world model from 188M to 430M parameters worsened performance on 4 of 4 suites, so capacity alone does not help. Second, the training objective matters far more: adding an auxiliary supervised contrastive (InfoNCE) loss during world-model warmup substantially improves sleep-phase clustering, giving Normalized Mutual Information at n=3 seeds of 0.462 ± 0.021 (object), 0.867 ± 0.025 (spatial), 0.915 ± 0.013 (goal) and 0.754 ± 0.010 (LIBERO-10), and beating the strongest published baseline on all 4 suites by a mean Delta = +0.184. Across providers (n=12) the 95% bootstrap confidence interval for mean pairwise NMI is [0.683, 0.729] (mean 0.705). The sleep phase also yields the first real-LIBERO task-language library: the decoder uses 2 of 3 admitted abstractions and rewrites all 256 sampled demonstrations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

27 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 26

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2020Supervised Contrastive Learningcited by 2,829
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  8. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2019On Variational Bounds of Mutual Informationcited by 559
  11. 2016The Option-Critic Architecturecited by 448
  12. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
14 more
  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  3. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  4. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  5. 2022Top-Down Synthesis for Library Learningcited by 58
  6. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  7. 2024Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Modelscited by 27
  8. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  9. 2026AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robotscited by 16
  10. 2024Hierarchical State Space Models for Continuous Sequence-to-Sequence Modelingcited by 15
  11. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  12. 2018CompILE: Compositional Imitation Learning and Executioncited by 7
  13. 2025Learning Semantic Atomic Skills for Multi-Task Robotic Manipulationcited by 5
  14. 2021Data-driven model predictive control: closed-loop guarantees and experimental resultscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VLA-Forget: Vision-Language-Action Unlearning for Embodied Foundation Models
  2. 2026Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Model
  3. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Model
  4. 2026ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models
  5. 2026rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference
  6. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Action
  7. 2026ActionGround: Training-Free Runtime Refinement of Frozen VLA Policies
  8. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines