# Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generation

arXiv: https://arxiv.org/abs/2211.15603 · HTML: https://mlsanity.com/paper/2211.15603/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [MUGL: Large Scale Multi Person Conditional Action Generation with Locomotion](https://mlsanity.com/paper/2110.11460/) (2021)

## Built on by (most cited first, 28 of 28)

- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [Inter-X: Towards Versatile Human-Human Interaction Analysis](https://mlsanity.com/paper/2312.16051/) (2023)
- [CoMo: Controllable Motion Generation through Language Guided Pose Code Editing](https://mlsanity.com/paper/2403.13900/) (2024)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [Generating Human Motion in 3D Scenes from Text Descriptions](https://mlsanity.com/paper/2405.07784/) (2024)
- [MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities](https://mlsanity.com/paper/2504.02478/) (2025)
- [MotionScript: Natural Language Descriptions for Expressive 3D Human Motions](https://mlsanity.com/paper/2312.12634/) (2023)
- [Fg-T2M++: LLMs-Augmented Fine-Grained Text Driven Human Motion Generation](https://mlsanity.com/paper/2502.05534/) (2025)
- [Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models](https://mlsanity.com/paper/2305.12356/) (2023)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [DiffusionPhase: Motion Diffusion in Frequency Domain](https://mlsanity.com/paper/2312.04036/) (2023)
- [MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations](https://mlsanity.com/paper/2410.13790/) (2024)
- [Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning](https://mlsanity.com/paper/2405.20606/) (2024)
- [MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion](https://mlsanity.com/paper/2409.12140/) (2024)
- [FineMotion: A Dataset and Benchmark with both Spatial and Temporal Annotation for Fine-grained Motion Generation and Editing](https://mlsanity.com/paper/2507.19850/) (2025)
- [Text-driven Motion Generation: Overview, Challenges and Directions](https://mlsanity.com/paper/2505.09379/) (2025)
- [FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions](https://mlsanity.com/paper/2312.02772/) (2023)
- [FineXtrol: Controllable Motion Generation via Fine-Grained Text](https://mlsanity.com/paper/2511.18927/) (2025)
- [Aligning Actions and Walking to LLM-Generated Textual Descriptions](https://mlsanity.com/paper/2404.12192/) (2024)
- [CigTime: Corrective Instruction Generation Through Inverse Motion Editing](https://mlsanity.com/paper/2412.05460/) (2024)
- [Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model](https://mlsanity.com/paper/2409.09276/) (2024)
- [Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2602.10659/) (2026)
- [Talk is Not Always Cheap: Promoting Wireless Sensing Models with Text Prompts](https://mlsanity.com/paper/2504.14621/) (2025)
- [LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference](https://mlsanity.com/paper/2603.11605/) (2026)
- [Generative Sign-description Prompts with Multi-positive Contrastive Learning for Sign Language Recognition](https://mlsanity.com/paper/2505.02304/) (2025)
- [MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation](https://mlsanity.com/paper/2605.18956/) (2026)
- [In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics](https://mlsanity.com/paper/2606.26981/) (2026)
- [Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation](https://mlsanity.com/paper/2608.23279/) (2026)

## Abstract (arXiv)

We introduce Action-GPT, a plug-and-play framework for incorporating Large Language Models (LLMs) into text-based action generation models. Action phrases in current motion capture datasets contain minimal and to-the-point information. By carefully crafting prompts for LLMs, we generate richer and fine-grained descriptions of the action. We show that utilizing these detailed descriptions instead of the original action phrases leads to better alignment of text and motion spaces. We introduce a generic approach compatible with stochastic (e.g. VAE-based) and deterministic (e.g. MotionCLIP) text-to-motion models. In addition, the approach enables multiple text descriptions to be utilized. Our experiments show (i) noticeable qualitative and quantitative improvement in the quality of synthesized motions, (ii) benefits of utilizing multiple LLM-generated descriptions, (iii) suitability of the prompt function, and (iv) zero-shot generation capabilities of the proposed approach. Project page: https://actiongpt.github.io