mlsanityData to 7 Oct 2026

Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)

arXiv 2307.09072 ↗First posted 18 Jul 2023cs.LG cs.AI cs.CEcited by 16

Not on the map. Why?

Abstract (arXiv)

Extrapolation remains a grand challenge in deep neural networks across all application domains. We propose an operator learning method to solve time-dependent partial differential equations (PDEs) continuously and with extrapolation in time without any temporal discretization. The proposed method, named Diffusion-inspired Temporal Transformer Operator (DiTTO), is inspired by latent diffusion models and their conditioning mechanism, which we use to incorporate the temporal evolution of the PDE, in combination with elements from the transformer architecture to improve its capabilities. Upon training, DiTTO can make inferences in real-time. We demonstrate its extrapolation capability on a climate problem by estimating the temperature around the globe for several years, and also in modeling hypersonic flows around a double-cone. We propose different training strategies involving temporal-bundling and sub-sampling and demonstrate performance improvements for several benchmarks, performing extrapolation for long time intervals as well as zero-shot super-resolution in time.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

36 of 64 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016Wide Residual Networkscited by 3,057
  10. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  11. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  12. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
24 more
  1. 2020A Survey on Visual Transformercited by 653
  2. 2022Transformers in Time Series: A Surveycited by 489
  3. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 470
  4. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  5. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  6. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  7. 2022Message Passing Neural PDE Solverscited by 368
  8. 2021Choose a Transformer: Fourier or Galerkincited by 333
  9. 2023ClimaX: A foundation model for weather and climatecited by 325
  10. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  11. 2017PDE-Net: Learning PDEs from Datacited by 304
  12. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  13. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  14. 2022U-NO: U-shaped Neural Operatorscited by 219
  15. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  16. 2022A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstructioncited by 150
  17. 2022Learning Operators with Coupled Attentioncited by 88
  18. 2022Learning two-phase microstructure evolution using neural operators and autoencoder architecturescited by 72
  19. 2022Blending Neural Operators and Relaxation Methods in PDE Numerical Solverscited by 53
  20. 2023LNO: Laplace Neural Operator for Solving Differential Equationscited by 47
  21. 2019DL-PDE: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy datacited by 38
  22. 2023ViTO: Vision Transformer-Operatorcited by 29
  23. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  24. 2023Learning bias corrections for climate models using deep neural operatorscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 5; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 32
  2. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 29
  3. 2024RiemannONets: Interpretable Neural Operators for Riemann Problemscited by 28
  4. 2023Rethinking materials simulations: Blending direct numerical simulations with neural operatorscited by 25
  5. 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 19
  6. 2024Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Modelscited by 14
  7. 2025Fusion-DeepONet: A Data-Efficient Neural Operator for Geometry-Dependent Hypersonic and Supersonic Flowscited by 13
  8. 2024Accelerating Phase Field Simulations Through a Hybrid Adaptive Fourier Neural Operator with U-Net Backbonecited by 10
  9. 2023Scattering with Neural Operatorscited by 7
  10. 2024Ricci flow regularization in latent spaces for the forward learning of partial differential equationscited by 2
  11. 2024PIC2O-Sim: A Physics-Inspired Causality-Aware Dynamic Convolutional Neural Operator for Ultra-Fast Photonic Device FDTD Simulationcited by 2
  12. 2025Extrapolating Phase-Field Simulations in Space and Time with Purely Convolutional Architecturescited by 1
4 more
  1. 2024Variational autoencoders with latent high-dimensional steady geometric flows for dynamicscited by 0
  2. 2025Learning Mappings in Mesh-based Simulationscited by 0
  3. 2025Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEscited by 0
  4. 2026Towards Spatio-Temporal Extrapolation of Phase-Field Simulations with Convolution-Only Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CFO: Learning Continuous-Time PDE Dynamics via Flow-Matched Neural Operators
  2. 2024Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems
  3. 2026Learning Physical Operators using Neural Operators
  4. 2022Fast Sampling of Diffusion Models via Operator Learning
  5. 2025Generative Latent Neural PDE Solver using Flow Matching
  6. 2026Autoregression-Free Neural Operators for Time-Dependent PDEs
  7. 2025Merging Memory and Space: A State Space Neural Operator
  8. 2026COSMOS: Soft Mechanism Mixtures with Verifiable Routing for Long-Horizon PDE Forecasting