# Generalizable Spacecraft Trajectory Generation via Multimodal Learning with Transformers

arXiv: https://arxiv.org/abs/2410.11723 · HTML: https://mlsanity.com/paper/2410.11723/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Transformers for Trajectory Optimization with Application to Spacecraft Rendezvous](https://mlsanity.com/paper/2310.13831/) (2023)
- [A Survey on Artificial Intelligence Trends in Spacecraft Guidance Dynamics and Control](https://mlsanity.com/paper/1812.02948/) (2018)
- [Towards Robust Spacecraft Trajectory Optimization via Transformers](https://mlsanity.com/paper/2410.05585/) (2024)
- [Bridging the Domain Gap for Flight-Ready Spaceborne Vision](https://mlsanity.com/paper/2409.11661/) (2024)
- [Advances in Trajectory Optimization for Space Vehicle Control](https://mlsanity.com/paper/2108.02335/) (2021)
- [Differential Dynamic Programming with Nonlinear Safety Constraints Under System Uncertainties](https://mlsanity.com/paper/2011.01051/) (2020)

## Built on by (most cited first, 6 of 6)

- [Deep Learning Warm Starts for Trajectory Optimization on the International Space Station](https://mlsanity.com/paper/2505.05588/) (2025)
- [Warm-Starting Optimization-Based Motion Planning for Robotic Manipulators via Point Cloud-Conditioned Flow Matching](https://mlsanity.com/paper/2510.03460/) (2025)
- [Agile Tradespace Exploration for Space Rendezvous Mission Design via Transformers](https://mlsanity.com/paper/2510.03544/) (2025)
- [Language-Conditioned Safe Trajectory Generation for Spacecraft Rendezvous](https://mlsanity.com/paper/2512.09111/) (2025)
- [On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning](https://mlsanity.com/paper/2604.07944/) (2026)
- [Transformer-Informed Trajectory Optimization for Relative Motion in Cislunar Orbits](https://mlsanity.com/paper/2609.25460/) (2026)

## Abstract (arXiv)

Effective trajectory generation is essential for reliable on-board spacecraft autonomy. Among other approaches, learning-based warm-starting represents an appealing paradigm for solving the trajectory generation problem, effectively combining the benefits of optimization- and data-driven methods. Current approaches for learning-based trajectory generation often focus on fixed, single-scenario environments, where key scene characteristics, such as obstacle positions or final-time requirements, remain constant across problem instances. However, practical trajectory generation requires the scenario to be frequently reconfigured, making the single-scenario approach a potentially impractical solution. To address this challenge, we present a novel trajectory generation framework that generalizes across diverse problem configurations, by leveraging high-capacity transformer neural networks capable of learning from multimodal data sources. Specifically, our approach integrates transformer-based neural network models into the trajectory optimization process, encoding both scene-level information (e.g., obstacle locations, initial and goal states) and trajectory-level constraints (e.g., time bounds, fuel consumption targets) via multimodal representations. The transformer network then generates near-optimal initial guesses for non-convex optimization problems, significantly enhancing convergence speed and performance. The framework is validated through extensive simulations and real-world experiments on a free-flyer platform, achieving up to 30% cost improvement and 80% reduction in infeasible cases with respect to traditional approaches, and demonstrating robust generalization across diverse scenario variations.