# Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

arXiv: https://arxiv.org/abs/2305.01661 · HTML: https://mlsanity.com/paper/2305.01661/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 37 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation](https://mlsanity.com/paper/2010.01114/) (2020)
- [Three Steps to Multimodal Trajectory Prediction: Modality Clustering, Classification and Synthesis](https://mlsanity.com/paper/2103.07854/) (2021)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Probabilistic Multi-modal Trajectory Prediction with Lane Attention for Autonomous Vehicles](https://mlsanity.com/paper/2007.02574/) (2020)
- [ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems](https://mlsanity.com/paper/2102.08535/) (2021)
- [A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approaches](https://mlsanity.com/paper/2111.02041/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Graph-Enhanced Deep-Reinforcement Learning Framework for the Aircraft Landing Problem](https://mlsanity.com/paper/2502.12617/) (2025)
- [Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing](https://mlsanity.com/paper/2511.12630/) (2025)

## Abstract (arXiv)

The booming air transportation industry inevitably burdens air traffic controllers' workload, causing unexpected human factor-related incidents. Current air traffic control systems fail to consider spoken instructions for traffic prediction, bringing significant challenges in detecting human errors during real-time traffic operations. Here, we present an automation paradigm integrating controlling intent into the information processing loop through the spoken instruction-aware flight trajectory prediction framework. A 3-stage progressive multi-modal learning paradigm is proposed to address the modality gap between the trajectory and spoken instructions, as well as minimize the data requirements. Experiments on a real-world dataset show the proposed framework achieves flight trajectory prediction with high predictability and timeliness, obtaining over 20% relative reduction in mean deviation error. Moreover, the generalizability of the proposed framework is also confirmed by various model architectures. The proposed framework can formulate full-automated information processing in real-world air traffic applications, supporting human error detection and enhancing aviation safety.