# Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning

arXiv: https://arxiv.org/abs/2411.14519 · HTML: https://mlsanity.com/paper/2411.14519/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2024.

## Built on by (most cited first, 19 of 19)

- [Motus: A Unified Latent Action World Model](https://mlsanity.com/paper/2512.13030/) (2025)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers](https://mlsanity.com/paper/2507.01016/) (2025)
- [CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning](https://mlsanity.com/paper/2505.17006/) (2025)
- [3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks](https://mlsanity.com/paper/2505.05800/) (2025)
- [AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots](https://mlsanity.com/paper/2603.07648/) (2026)
- [Expertise need not monopolize: Action-Specialized Mixture of Experts for Vision-Language-Action Learning](https://mlsanity.com/paper/2510.14300/) (2025)
- [What Happens Next? Anticipating Future Motion by Generating Point Trajectories](https://mlsanity.com/paper/2509.21592/) (2025)
- [MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding](https://mlsanity.com/paper/2511.22103/) (2025)
- [Forecasting Motion in the Wild](https://mlsanity.com/paper/2604.01015/) (2026)
- [From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation](https://mlsanity.com/paper/2605.12167/) (2026)
- [HMVLA: Hyperbolic Multimodal Fusion for Vision-Language-Action Models](https://mlsanity.com/paper/2602.02533/) (2026)
- [TrajLoom: Dense Future Trajectory Generation from Video](https://mlsanity.com/paper/2603.22606/) (2026)
- [Learning Long-term Motion Embeddings for Efficient Kinematics Generation](https://mlsanity.com/paper/2604.11737/) (2026)
- [Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Training](https://mlsanity.com/paper/2607.19971/) (2026)
- [SceneSelect: Selective Learning for Trajectory Scene Classification and Expert Scheduling](https://mlsanity.com/paper/2604.24514/) (2026)
- [3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints](https://mlsanity.com/paper/2609.08224/) (2026)
- [NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields](https://mlsanity.com/paper/2610.00981/) (2026)
- [JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image](https://mlsanity.com/paper/2610.16591/) (2026)

## Abstract (arXiv)

Learning from multiple domains is a primary factor that influences the generalization of a single unified robot system. In this paper, we aim to learn the trajectory prediction model by using broad out-of-domain data to improve its performance and generalization ability. Trajectory model is designed to predict any-point trajectories in the current frame given an instruction and can provide detailed control guidance for robotic policy learning. To handle the diverse out-of-domain data distribution, we propose a sparsely-gated MoE (Top-1 gating strategy) architecture for trajectory model, coined as Tra-MoE. The sparse activation design enables good balance between parameter cooperation and specialization, effectively benefiting from large-scale out-of-domain data while maintaining constant FLOPs per token. In addition, we further introduce an adaptive policy conditioning technique by learning 2D mask representations for predicted trajectories, which is explicitly aligned with image observations to guide action prediction more flexibly. We perform extensive experiments on both simulation and real-world scenarios to verify the effectiveness of Tra-MoE and adaptive policy conditioning technique. We also conduct a comprehensive empirical study to train Tra-MoE, demonstrating that our Tra-MoE consistently exhibits superior performance compared to the dense baseline model, even when the latter is scaled to match Tra-MoE's parameter count.