mlsanityData to 14 Oct 2026

HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models

arXiv 2503.19157 ↗First posted 24 Mar 2025cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (eg text, objects, partial sequences). At its core, HOIGPT utilizes a large language model to predict the bidrectional transformation between HOI sequences and natural language descriptions. Given text inputs, HOIGPT generates a sequence of hand and object meshes; given (partial) HOI sequences, HOIGPT generates text descriptions and completes the sequences. To facilitate HOI understanding with a large language model, this paper introduces two key innovations: (1) a novel physically grounded HOI tokenizer, the hand-object decomposed VQ-VAE, for discretizing HOI sequences, and (2) a motion-aware language model trained to process and generate both text and HOI tokens. Extensive experiments demonstrate that HOIGPT sets new state-of-the-art performance on both text generation (+2.01% R Precision) and HOI generation (-2.56 FID) across multiple tasks and benchmarks.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 46 of its 52 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

46 of 52 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
34 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  3. 2022Human Motion Diffusion Modelcited by 1,080
  4. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  7. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  8. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  9. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  10. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  11. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  13. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
  14. 2023Human Motion Diffusion as a Generative Priorcited by 357
  15. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  16. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 185
  17. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 168
  18. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  19. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  20. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  21. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  22. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 117
  23. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  24. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  25. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  26. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 86
  27. 2023Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Modelcited by 79
  28. 2024Flexible Motion In-betweening with Diffusion Modelscited by 71
  29. 2024Humanoid Locomotion as Next Token Predictioncited by 69
  30. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 62
  31. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  32. 2023Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generationcited by 56
  33. 2023DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusioncited by 37
  34. 2023Language-guided Human Motion Synthesis with Atomic Actionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 24.

33 papers.

Sort:

By area:

  1. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  2. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 22
  3. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 18
  4. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  5. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  6. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 12
  7. 2025EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observationscited by 9
  8. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  9. 2025Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representationcited by 5
  10. 2026GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splattingcited by 3
  11. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  12. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
21 more
  1. 2026HandX: Scaling Bimanual Motion and Interaction Generationcited by 2
  2. 2026Dex2HOI: Dexterous Bimanual Two-Object Interaction Generationcited by 2
  3. 2026JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generationcited by 2
  4. 2026Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generationcited by 2
  5. 2026HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generationcited by 2
  6. 2025SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representationcited by 1
  7. 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
  8. 2026GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awarenesscited by 1
  9. 2026HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matchingcited by 1
  10. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  11. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1
  12. 2026DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Targetcited by 0
  13. 2026OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesiscited by 0
  14. 2026DuetHOI: Language-Guided Bimanual Hand--Object Motion Generation with Articulation Planning and Contact Refinementcited by 0
  15. 2026RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signalscited by 0
  16. 2026PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photographcited by 0
  17. 2026DigitCode: Symbolic Tokenization of Hand Motion by Anatomical Unitscited by 0
  18. 2026MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Textcited by 0
  19. 2026EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecastingcited by 0
  20. 2026Harnessing Coupled Stream Completion For Human-Object Interaction Modelingcited by 0
  21. 2026MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlowcited by 0