mlsanityData to 14 Oct 2026

Humanoid Locomotion as Next Token Prediction

arXiv 2402.19469 ↗First posted 29 Feb 2024cs.RO cs.CV cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

25 of 44 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
13 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  6. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  7. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  8. 2016The KIT Motion-Language Datasetcited by 338
  9. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  10. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 116
  11. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  12. 2021The MIT Humanoid Robot: Design, Motion Planning, and Control For Acrobatic Behaviorscited by 32
  13. 2021Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robotcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 32; 2026: 12.

69 papers.

Sort:

By area:

  1. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  2. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  3. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  4. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
  5. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 130
  6. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  7. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
  8. 2024In-Context Imitation Learning via Next-Token Predictioncited by 70
  9. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 68
  10. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  11. 2024Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfercited by 53
  12. 2024Generalizable Humanoid Manipulation with 3D Diffusion Policiescited by 52
38 more
  1. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  2. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 46
  3. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  4. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 41
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  7. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 33
  8. 2024Body Transformer: Leveraging Robot Embodiment for Policy Learningcited by 32
  9. 2024Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policiescited by 30
  10. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 29
  11. 2025HuB: Learning Extreme Humanoid Balancecited by 28
  12. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  13. 2024Learning Humanoid Locomotion over Challenging Terraincited by 26
  14. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 26
  15. 2025Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulationcited by 25
  16. 2025Reconstructing Humans with a Biomechanically Accurate Skeletoncited by 22
  17. 2024RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewardscited by 18
  18. 2025From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robotscited by 17
  19. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  20. 2025Learning Humanoid Locomotion with World Model Reconstructioncited by 13
  21. 2026Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive sciencecited by 11
  22. 2025Physical Plausibility-aware Trajectory Prediction via Locomotion Embodimentcited by 10
  23. 2024Synergy and Synchrony in Couple Dancescited by 9
  24. 2025Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysiscited by 9
  25. 2025SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blendingcited by 9
  26. 2025LookOut: Real-World Humanoid Egocentric Navigationcited by 9
  27. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  28. 2026HoloMotion-1 Technical Reportcited by 8
  29. 2024Deep Reinforcement Learning for Bipedal Locomotion: A Brief Surveycited by 7
  30. 2025Embrace Collisions: Humanoid Shadowing for Deployable Contact-Agnostics Motionscited by 7
  31. 2025Natural Humanoid Robot Locomotion with Generative Motion Priorcited by 7
  32. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  33. 2025JAEGER: Dual-Level Humanoid Whole-Body Controllercited by 6
  34. 2025From Generated Human Videos to Physically Plausible Robot Trajectoriescited by 6
  35. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  36. 2025LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videoscited by 5
  37. 2025DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotioncited by 5
  38. 2025Understanding the Generalization of In-Context Learning in Transformers: An Empirical Studycited by 4

50 most cited of 69 All 69 (JSON)