# Learning Expressive and Compositional Motion Representation via Spectral Skills

arXiv: https://arxiv.org/abs/2609.37677 · HTML: https://mlsanity.com/paper/2609.37677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 45)

- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2307.12868/) (2023)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control](https://mlsanity.com/paper/2511.07820/) (2025)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots](https://mlsanity.com/paper/2410.21229/) (2024)
- [GMT: General Motion Tracking for Humanoid Whole-Body Control](https://mlsanity.com/paper/2506.14770/) (2025)
- [The Riemannian Geometry of Deep Generative Models](https://mlsanity.com/paper/1711.08014/) (2017)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning](https://mlsanity.com/paper/2501.02116/) (2025)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Making Linear MDPs Practical via Contrastive Representation Learning](https://mlsanity.com/paper/2207.07150/) (2022)
- [LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction](https://mlsanity.com/paper/2506.13751/) (2025)
- [Object-Centric Dexterous Manipulation from Human Motion Data](https://mlsanity.com/paper/2411.04005/) (2024)
- [CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations](https://mlsanity.com/paper/2505.04999/) (2025)
- [BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2511.04131/) (2025)
- [UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots](https://mlsanity.com/paper/2507.07356/) (2025)

## Abstract (arXiv)

Robotic foundation models offer a promising path toward general-purpose humanoid robot control, often through hierarchical architectures. However, their effectiveness depends on the command interface between the planner and the controller, which must support accurate execution while remaining easy to predict, and ideally allow new behaviors to be composed from prior ones. In this work, we introduce spectral skills, a latent representation of this interface that meets these requirements through predictive representation learning. By design, spectral skills compactly encode short motion segments and are learned by predicting subsequent motion rather than reconstructing the encoder input. On a 29-DoF humanoid, a controller conditioned on spectral skills reduces global tracking error by 62% relative to the state of the art. The same frozen controller chains independently encoded skills without a separate transition policy. It also composes new behaviors by adding orthogonal directions to any compatible base skill, producing combinations unseen in the training data. We demonstrate tracking, chaining, and composition, as well as control through a language-conditioned planner, on Unitree G1 hardware. Project page: https://spectral-skill.github.io