# Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

arXiv: https://arxiv.org/abs/2606.30266 · HTML: https://mlsanity.com/paper/2606.30266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 29 of 29)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding](https://mlsanity.com/paper/2410.21747/) (2024)
- [Self-Supervised Training Enhances Online Continual Learning](https://mlsanity.com/paper/2103.14010/) (2021)
- [MotionGPT3: Human Motion as a Second Modality](https://mlsanity.com/paper/2506.24086/) (2025)
- [Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning](https://mlsanity.com/paper/2506.11672/) (2025)
- [iManip: Skill-Incremental Learning for Robotic Manipulation](https://mlsanity.com/paper/2503.07087/) (2025)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [Parameter-Efficient Continual Fine-Tuning: A Survey](https://mlsanity.com/paper/2504.13822/) (2025)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)

## Abstract (arXiv)

Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M). While foundational models have achieved strong performance in static settings, autonomous agents operating in dynamic environments must continuously incorporate new motion concepts -- such as novel athletic styles or specialized gestures -- without catastrophic forgetting of previously acquired skills. We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure. Building on a frozen large language model backbone, we introduce low-rank adaptation (LoRA) variants designed to mitigate inter-task interference. We specifically propose mixture-of-experts architectures that utilize an autoencoder-based router to select task-specific experts at inference time, so that no task-label is needed. To evaluate these methods, we establish a reproducible five-task benchmark derived from HumanML3D through semantic clustering of motion descriptions. Our experimental results demonstrate near-zero forgetting across both M2T and T2M directions while maintaining high generation and captioning quality. Furthermore, we show that hard expert selection via routing significantly outperforms soft expert blending in quality metrics, indicating that preserving expert isolation is critical for maintaining performance in our continual learning setting. Finally, we observe that a divergence between token-level accuracy and downstream generation quality may occur, highlighting the need for more comprehensive evaluation protocols in future research on lifelong motion-language agents.