mlsanityData to 7 Oct 2026

A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics

arXiv 2604.15215 ↗First posted 16 Apr 2026cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, the lower level assigns input actions to fine-grained subclusters, while the higher level further maps fine-grained subclusters to clusters. Our hierarchical approach outperforms the non-hierarchical counterpart, while mainly exploiting spatial information by reconstructing input actions. Furthermore, we extend our approach by utilizing both spatial and temporal cues, forming a hierarchical spatiotemporal action tokenizer, namely HiST-AT. Specifically, our hierarchical spatiotemporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps. Finally, extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 53 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

50 of 53 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Action Tokenizer Matters in In-Context Imitation Learning3 Mar
  2. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation24 Jan
  3. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusion19 Nov
  4. 2024In-Context Imitation Learning via Next-Token Prediction28 Aug
  5. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics28 Mar
  6. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations26 Oct
  7. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignment18 Oct
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  9. 2022Behavior Transformers: Cloning k modes with one stone22 Jun

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
29 more
  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  3. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  4. 2023Large Language Models as General Pattern Machinescited by 196
  5. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  6. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  7. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  8. 2019Learning Action Representations for Reinforcement Learningcited by 75
  9. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  10. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  11. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  12. 2021Unsupervised Action Segmentation by Joint Representation Learning and Online Clusteringcited by 48
  13. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 47
  14. 2020Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequencescited by 45
  15. 2024Temporally Consistent Unbalanced Optimal Transport for Unsupervised Action Segmentationcited by 37
  16. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  17. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  18. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  19. 2024Hierarchical Vector Quantization for Unsupervised Action Segmentationcited by 18
  20. 2023Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignmentcited by 17
  21. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 11
  22. 2025Joint Self-Supervised Video Alignment and Action Segmentationcited by 11
  23. 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
  24. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  25. 2018Action Representations in Robotics: A Taxonomy and Systematic Classificationcited by 9
  26. 2024EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learningcited by 6
  27. 2025Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentationcited by 6
  28. 2023BeBOP -- Combining Reactive Planning and Bayesian Optimization to Solve Robotic Manipulation Taskscited by 5
  29. 2025Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
  2. 2026ST-π: Structured SpatioTemporal VLA for Robotic Manipulation
  3. 2026Mimic Intent, Not Just Trajectories
  4. 2026Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning
  5. 2026WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
  6. 2025Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Space
  7. 2026ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models
  8. 2025MOVE: A Simple Motion-Based Data Collection Paradigm for Spatial Generalization in Robotic Manipulation