# Multi-Modal Drift Forecasting of Leeway Objects via Navier-Stokes-Guided CNN and Sequence-to-Sequence Attention-Based Models

arXiv: https://arxiv.org/abs/2508.18284 · HTML: https://mlsanity.com/paper/2508.18284/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 61 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities](https://mlsanity.com/paper/2402.10835/) (2024)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [A DNN Framework for Learning Lagrangian Drift With Uncertainty](https://mlsanity.com/paper/2204.05891/) (2022)

## Abstract (arXiv)

Accurately predicting the drift (displacement) of leeway objects in maritime environments remains a critical challenge, particularly in time-sensitive scenarios such as search and rescue operations. In this study, we propose a multi-modal machine learning framework that integrates Sentence Transformer embeddings with attention-based sequence-to-sequence architectures to predict the drift of leeway objects in water. We begin by experimentally collecting environmental and physical data, including water current and wind velocities, object mass, and surface area, for five distinct leeway objects. Using simulated data from a Navier-Stokes-based model to train a convolutional neural network on geometrical image representations, we estimate drag and lift coefficients of the leeway objects. These coefficients are then used to derive the net forces responsible for driving the objects' motion. The resulting time series, comprising physical forces, environmental velocities, and object-specific features, combined with textual descriptions encoded via a language model, are inputs to attention-based sequence-to-sequence long-short-term memory and Transformer models, to predict future drift trajectories. We evaluate the framework across multiple time horizons (1, 3, 5, and 10 seconds) and assess its generalization across different objects. We compare our approach against a fitted physics-based model and traditional machine learning methods, including recurrent neural networks and temporal convolutional neural networks. Our results show that these multi-modal models perform comparably to traditional models while also enabling longer-term forecasting in place of single-step prediction. Overall, our findings demonstrate the ability of a multi-modal modeling strategy to provide accurate and adaptable predictions of leeway object drift in dynamic maritime conditions.