# CViT: Continuous Vision Transformer for Operator Learning

arXiv: https://arxiv.org/abs/2405.13998 · HTML: https://mlsanity.com/paper/2405.13998/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 80 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)

## Built on by (most cited first, 30 of 51)

- [FunDiff: Diffusion Models over Function Spaces for Physics-Informed Generative Modeling](https://mlsanity.com/paper/2506.07902/) (2025)
- [Learning Enhanced Ensemble Filters](https://mlsanity.com/paper/2504.17836/) (2025)
- [Mesh-Informed Neural Operator : A Transformer Generative Approach](https://mlsanity.com/paper/2506.16656/) (2025)
- [Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers](https://mlsanity.com/paper/2507.07814/) (2025)
- [CFO: Learning Continuous-Time PDE Dynamics via Flow-Matched Neural Operators](https://mlsanity.com/paper/2512.05297/) (2025)
- [TANTE: Time-Adaptive Operator Learning via Neural Taylor Expansion](https://mlsanity.com/paper/2502.08574/) (2025)
- [PhysicsCorrect: A Training-Free Approach for Stable Neural PDE Simulations](https://mlsanity.com/paper/2507.02227/) (2025)
- [Structure-Preserving Digital Twins via Conditional Neural Whitney Forms](https://mlsanity.com/paper/2508.06981/) (2025)
- [Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs](https://mlsanity.com/paper/2602.02788/) (2026)
- [Geometry aware inference of steady state PDEs using Equivariant Neural Fields representations](https://mlsanity.com/paper/2504.18591/) (2025)
- [Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics](https://mlsanity.com/paper/2507.02748/) (2025)
- [Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators](https://mlsanity.com/paper/2507.09264/) (2025)
- [Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Model](https://mlsanity.com/paper/2412.13897/) (2024)
- [MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention](https://mlsanity.com/paper/2512.01738/) (2025)
- [Adaptive Mesh-Quantization for Neural PDE Solvers](https://mlsanity.com/paper/2511.18474/) (2025)
- [Efficient Generative Transformer Operators For Million-Point PDEs](https://mlsanity.com/paper/2512.04974/) (2025)
- [Function graph transformers universally approximate operators between function spaces](https://mlsanity.com/paper/2605.17968/) (2026)
- [Towards scientific machine learning for granular material simulations -- challenges and opportunities](https://mlsanity.com/paper/2504.08766/) (2025)
- [FA-INR: Adaptive Implicit Neural Representations for Interpretable Exploration of Simulation Ensembles](https://mlsanity.com/paper/2506.06858/) (2025)
- [Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator](https://mlsanity.com/paper/2510.16816/) (2025)
- [Discontinuous Galerkin finite element operator network for solving non-smooth PDEs](https://mlsanity.com/paper/2601.03668/) (2026)
- [Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems](https://mlsanity.com/paper/2602.01486/) (2026)
- [Adaptive Physics Transformer with Fused Global-Local Attention for Subsurface Energy Systems](https://mlsanity.com/paper/2602.11208/) (2026)
- [Physics-Informed Neural PDE Solvers via Spatio-Temporal MeanFlow](https://mlsanity.com/paper/2605.08915/) (2026)
- [Towards a Foundation Model for the Martian Atmosphere](https://mlsanity.com/paper/2605.28851/) (2026)
- [On the training of physics-informed neural operators for solving parametric partial differential equations](https://mlsanity.com/paper/2606.06164/) (2026)
- [From Points to Edges: Edge-Conditioned Spectral Operators for Physics-Sensitive PDE Learning](https://mlsanity.com/paper/2608.06894/) (2026)
- [TemporalFlowViz: Parameter-Aware Visual Analytics for Interpreting Scramjet Combustion Evolution](https://mlsanity.com/paper/2509.04834/) (2025)
- [GeoFunFlow: Geometric Function Flow Matching for Inverse Operator Learning over Complex Geometries](https://mlsanity.com/paper/2509.24117/) (2025)
- [Axial Neural Networks for Dimension-Free Foundation Models](https://mlsanity.com/paper/2510.13665/) (2025)

## Abstract (arXiv)

Operator learning, which aims to approximate maps between infinite-dimensional function spaces, is an important area in scientific machine learning with applications across various physical domains. Here we introduce the Continuous Vision Transformer (CViT), a novel neural operator architecture that leverages advances in computer vision to address challenges in learning complex physical systems. CViT combines a vision transformer encoder, a novel grid-based coordinate embedding, and a query-wise cross-attention mechanism to effectively capture multi-scale dependencies. This design allows for flexible output representations and consistent evaluation at arbitrary resolutions. We demonstrate CViT's effectiveness across a diverse range of partial differential equation (PDE) systems, including fluid dynamics, climate modeling, and reaction-diffusion processes. Our comprehensive experiments show that CViT achieves state-of-the-art performance on multiple benchmarks, often surpassing larger foundation models, even without extensive pretraining and roll-out fine-tuning. Taken together, CViT exhibits robust handling of discontinuous solutions, multi-scale features, and intricate spatio-temporal dynamics. Our contributions can be viewed as a significant step towards adapting advanced computer vision architectures for building more flexible and accurate machine learning models in the physical sciences.