# SATO: Stable Text-to-Motion Framework

arXiv: https://arxiv.org/abs/2405.01461 · HTML: https://mlsanity.com/paper/2405.01461/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [Understanding and Mitigating the Tradeoff Between Robustness and Accuracy](https://mlsanity.com/paper/2002.10716/) (2020)
- [Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model](https://mlsanity.com/paper/2309.06284/) (2023)
- [Interpretable Adversarial Perturbation in Input Embedding Space for Text](https://mlsanity.com/paper/1805.02917/) (2018)
- [Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs](https://mlsanity.com/paper/2311.01015/) (2023)
- [Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings](https://mlsanity.com/paper/2210.01448/) (2022)
- [Fair Text-to-Image Diffusion via Fair Mapping](https://mlsanity.com/paper/2311.17695/) (2023)
- [On the Sensitivity and Stability of Model Interpretations in NLP](https://mlsanity.com/paper/2104.08782/) (2021)
- [VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders](https://mlsanity.com/paper/2309.01141/) (2023)
- [Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training](https://mlsanity.com/paper/2009.12064/) (2020)
- [Improving Interpretation Faithfulness for Vision Transformers](https://mlsanity.com/paper/2311.17983/) (2023)
- [Fine-tuning CLIP Text Encoders with Two-step Paraphrasing](https://mlsanity.com/paper/2402.15120/) (2024)

## Built on by (most cited first, 20 of 20)

- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [Physics-Informed Representation Alignment for Sparse Radio-Map Reconstruction](https://mlsanity.com/paper/2501.19160/) (2025)
- [ANT: Adaptive Neural Temporal-Aware Text-to-Motion Model](https://mlsanity.com/paper/2506.02452/) (2025)
- [CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation](https://mlsanity.com/paper/2508.03535/) (2025)
- [FTS: A Framework to Find a Faithful TimeSieve](https://mlsanity.com/paper/2405.19647/) (2024)
- [Towards Robust and Controllable Text-to-Motion via Masked Autoregressive Diffusion](https://mlsanity.com/paper/2505.11013/) (2025)
- [Free-T2M: Robust Text-to-Motion Generation for Humanoid Robots via Frequency-Domain](https://mlsanity.com/paper/2501.18232/) (2025)
- [AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios](https://mlsanity.com/paper/2511.21053/) (2025)
- [MemoGen: Can Past Experience Improve Future Text-to-Image Generation?](https://mlsanity.com/paper/2606.03243/) (2026)
- [Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner](https://mlsanity.com/paper/2407.18656/) (2024)
- [Oracle Noise: Faster Semantic Spherical Alignment for Interpretable Latent Optimization](https://mlsanity.com/paper/2604.23540/) (2026)
- [GenTrack: Physical Alignment for Robot-Native Motion Generation and Zero-Shot Humanoid Tracking](https://mlsanity.com/paper/2608.01410/) (2026)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Z2-Sampling: Zero-Cost Zigzag Trajectories for Semantic Alignment in Diffusion Models](https://mlsanity.com/paper/2604.23536/) (2026)
- [AI-Generated Fall Data: Assessing LLMs and Diffusion Model for Wearable Fall Detection](https://mlsanity.com/paper/2505.04660/) (2025)
- [ChairPose: Pressure-based Chair Morphology Grounded Sitting Pose Estimation through Simulation-Assisted Training](https://mlsanity.com/paper/2508.01850/) (2025)
- [A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniques](https://mlsanity.com/paper/2512.09005/) (2025)
- [T2M Mamba: Motion Periodicity-Saliency Coupling Approach for Stable Text-Driven Motion Generation](https://mlsanity.com/paper/2602.01352/) (2026)
- [EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challenges](https://mlsanity.com/paper/2604.22595/) (2026)
- [Before the Body Moves: Learning Anticipatory Joint Intent for Language-Conditioned Humanoid Control](https://mlsanity.com/paper/2605.14417/) (2026)

## Abstract (arXiv)

Is the Text to Motion model robust? Recent advancements in Text to Motion models primarily stem from more accurate predictions of specific actions. However, the text modality typically relies solely on pre-trained Contrastive Language-Image Pretraining (CLIP) models. Our research has uncovered a significant issue with the text-to-motion model: its predictions often exhibit inconsistent outputs, resulting in vastly different or even incorrect poses when presented with semantically similar or identical text inputs. In this paper, we undertake an analysis to elucidate the underlying causes of this instability, establishing a clear link between the unpredictability of model outputs and the erratic attention patterns of the text encoder module. Consequently, we introduce a formal framework aimed at addressing this issue, which we term the Stable Text-to-Motion Framework (SATO). SATO consists of three modules, each dedicated to stable attention, stable prediction, and maintaining a balance between accuracy and robustness trade-off. We present a methodology for constructing an SATO that satisfies the stability of attention and prediction. To verify the stability of the model, we introduced a new textual synonym perturbation dataset based on HumanML3D and KIT-ML. Results show that SATO is significantly more stable against synonyms and other slight perturbations while keeping its high accuracy performance.