# Spatially-Enhanced Temporal Fusion Transformer: Interpretable Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputs

arXiv: https://arxiv.org/abs/2505.00473 · HTML: https://mlsanity.com/paper/2505.00473/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs](https://mlsanity.com/paper/2001.04001/) (2020)
- [POD-DL-ROM: enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition](https://mlsanity.com/paper/2101.11845/) (2021)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version](https://mlsanity.com/paper/2204.13767/) (2022)
- [Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://mlsanity.com/paper/2206.04038/) (2022)
- [TACTiS: Transformer-Attentional Copulas for Time Series](https://mlsanity.com/paper/2202.03528/) (2022)
- [Quadratic Approximation Manifold for Mitigating the Kolmogorov Barrier in Nonlinear Projection-Based Model Order Reduction](https://mlsanity.com/paper/2204.02462/) (2022)
- [gLaSDI: Parametric Physics-informed Greedy Latent Space Dynamics Identification](https://mlsanity.com/paper/2204.12005/) (2022)
- [GPLaSDI: Gaussian Process-based Interpretable Latent Space Dynamics Identification through Deep Autoencoder](https://mlsanity.com/paper/2308.05882/) (2023)
- [Reduced order modeling of parametrized systems through autoencoders and SINDy approach: continuation of periodic solutions](https://mlsanity.com/paper/2211.06786/) (2022)
- [Multi-fidelity surrogate modeling using long short-term memory networks](https://mlsanity.com/paper/2208.03115/) (2022)
- [SSDNet: State Space Decomposition Neural Network for Time Series Forecasting](https://mlsanity.com/paper/2112.10251/) (2021)
- [A greedy non-intrusive reduced order model for shallow water equations](https://mlsanity.com/paper/2002.11329/) (2020)
- [TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting](https://mlsanity.com/paper/2108.12784/) (2021)
- [Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models](https://mlsanity.com/paper/2410.13376/) (2024)

## Built on by (most cited first, 1 of 1)

- [Target Parameterization in Diffusion Models for Nonlinear Spatiotemporal System Identification](https://mlsanity.com/paper/2604.17566/) (2026)

## Abstract (arXiv)

We explore the promising performance of a transformer model in predicting outputs of parametric dynamical systems with external time-varying input signals. The outputs of such systems vary not only with physical parameters but also with external time-varying input signals. Accurately catching the dynamics of such systems is challenging. We have adapted and extended an existing transformer model, called temporal fusion transformer (TFT), for single-output prediction to a multiple-output transformer, named as Spatially-Enhanced Temporal Fusion Transformer (SE-TFT), which is able to predict multiple output responses of these systems. The SE-TFT generalizes the interpretability of the original TFT model. The generalized interpretable attention weight matrix explores not only the temporal correlations in the sequence, but also the interactions between the multiple outputs, providing explanation for the spatial correlation in the output domain. This proposed SE-TFT accurately predicts the sequence of multiple outputs, regardless of the nonlinearity of the system and the dimensionality of the parameter space.