# ViTO: Vision Transformer-Operator

arXiv: https://arxiv.org/abs/2303.08891 · HTML: https://mlsanity.com/paper/2303.08891/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [DeepXDE: A deep learning library for solving differential equations](https://mlsanity.com/paper/1907.04502/) (2019)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration](https://mlsanity.com/paper/2104.06468/) (2021)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)

## Built on by (most cited first, 29 of 29)

- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Improved Operator Learning by Orthogonal Attention](https://mlsanity.com/paper/2310.12487/) (2023)
- [Continuum Attention for Neural Operators](https://mlsanity.com/paper/2406.06486/) (2024)
- [GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer](https://mlsanity.com/paper/2512.20399/) (2025)
- [Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)](https://mlsanity.com/paper/2307.09072/) (2023)
- [Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2510.25803/) (2025)
- [MyCrunchGPT: A chatGPT assisted framework for scientific machine learning](https://mlsanity.com/paper/2306.15551/) (2023)
- [Fast meta-solvers for 3D complex-shape scatterers using neural operators trained on a non-scattering problem](https://mlsanity.com/paper/2405.12380/) (2024)
- [Transformer learns the cross-task prior and regularization for in-context learning](https://mlsanity.com/paper/2505.12138/) (2025)
- [About optimal loss function for training physics-informed neural networks under respecting causality](https://mlsanity.com/paper/2304.02282/) (2023)
- [Separable Physics-Informed Neural Networks for the solution of elasticity problems](https://mlsanity.com/paper/2401.13486/) (2024)
- [Separable Operator Networks](https://mlsanity.com/paper/2407.11253/) (2024)
- [Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators](https://mlsanity.com/paper/2507.09264/) (2025)
- [Learning Causal Graphs at Scale: A Foundation Model Approach](https://mlsanity.com/paper/2506.18285/) (2025)
- [Physics-informed UNets for Discovering Hidden Elasticity in Heterogeneous Materials](https://mlsanity.com/paper/2306.01204/) (2023)
- [Brain Tumor Classification using Vision Transformer with Selective Cross-Attention Mechanism and Feature Calibration](https://mlsanity.com/paper/2406.17670/) (2024)
- [NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2602.22059/) (2026)
- [Understanding the Efficacy of U-Net & Vision Transformer for Groundwater Numerical Modelling](https://mlsanity.com/paper/2307.04010/) (2023)
- [ODE-DPS: ODE-based Diffusion Posterior Sampling for Inverse Problems in Partial Differential Equation](https://mlsanity.com/paper/2404.13496/) (2024)
- [Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systems](https://mlsanity.com/paper/2507.03660/) (2025)
- [FA-INR: Adaptive Implicit Neural Representations for Interpretable Exploration of Simulation Ensembles](https://mlsanity.com/paper/2506.06858/) (2025)
- [Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator](https://mlsanity.com/paper/2510.16816/) (2025)
- [In Transformer We Trust? A Perspective on Transformer Architecture Failure Modes](https://mlsanity.com/paper/2602.14318/) (2026)
- [Harnessing The Power of Attention For Patch-Based Biomedical Image Classification](https://mlsanity.com/paper/2404.00949/) (2024)
- [Prion-ViT: Prions-Inspired Vision Transformers for Temperature prediction with Specklegrams](https://mlsanity.com/paper/2411.05836/) (2024)
- [Skin Cancer Classification: Hybrid CNN-Transformer Models with KAN-Based Fusion](https://mlsanity.com/paper/2508.12484/) (2025)
- [Application of Graph Based Vision Transformers Architectures for Accurate Temperature Prediction in Fiber Specklegram Sensors](https://mlsanity.com/paper/2511.14792/) (2025)
- [Operator Learning for Reconstructing Flow Fields from Sparse Measurements: a Language Model Approach](https://mlsanity.com/paper/2605.23712/) (2026)
- [LLT: Local Linear Transformer for PDE Operator Learning](https://mlsanity.com/paper/2607.07718/) (2026)

## Abstract (arXiv)

We combine vision transformers with operator learning to solve diverse inverse problems described by partial differential equations (PDEs). Our approach, named ViTO, combines a U-Net based architecture with a vision transformer. We apply ViTO to solve inverse PDE problems of increasing complexity, namely for the wave equation, the Navier-Stokes equations and the Darcy equation. We focus on the more challenging case of super-resolution, where the input dataset for the inverse problem is at a significantly coarser resolution than the output. The results we obtain are comparable or exceed the leading operator network benchmarks in terms of accuracy. Furthermore, ViTO`s architecture has a small number of trainable parameters (less than 10% of the leading competitor), resulting in a performance speed-up of over 5x when averaged over the various test cases.