mlsanityData to 7 Oct 2026

Transformer Meets Boundary Value Inverse Problems

arXiv 2209.14977 ↗First posted 29 Sep 2022cs.LG cs.NA math.NAcited by 22

Not on the map. Why?

Abstract (arXiv)

A Transformer-based deep direct sampling method is proposed for electrical impedance tomography, a well-known severely ill-posed nonlinear boundary value inverse problem. A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images. An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks? Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels. Then, by introducing learnable non-local kernels, the direct sampling is recast to a modified attention mechanism. The new method achieves superior accuracy over its predecessors and contemporary operator learners and shows robustness to noises in benchmarks. This research shall strengthen the insights that, despite being invented for natural language processing tasks, the attention mechanism offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 59 of its 172 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

59 of 172 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
38 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 906
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
  4. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  5. 2016Deep Convolutional Neural Network for Inverse Problems in Imagingcited by 431
  6. 2021Random Feature Attentioncited by 350
  7. 2021Choose a Transformer: Fourier or Galerkincited by 333
  8. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 292
  9. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  10. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  11. 2017Image reconstruction by domain transform manifold learningcited by 231
  12. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 229
  13. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  14. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  15. 2022MIONet: Learning multiple-input operators via tensor productcited by 180
  16. 2021nnFormer: Interleaved Transformer for Volumetric Segmentationcited by 179
  17. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  18. 2021UTNet: A Hybrid Transformer Architecture for Medical Image Segmentationcited by 116
  19. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  20. 2021Mixed Transformer U-Net For Medical Image Segmentationcited by 95
  21. 2022Clifford Neural Layers for PDE Modelingcited by 91
  22. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  23. 2022Learning Operators with Coupled Attentioncited by 88
  24. 2016A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstructioncited by 87
  25. 2021Equivariant Imaging: Learning Beyond the Range Spacecited by 83
  26. 2021U-Net Transformer: Self and Cross Attention for Medical Image Segmentationcited by 81
  27. 2022NOMAD: Nonlinear Manifold Decoders for Operator Learningcited by 81
  28. 2022Generic bounds on the approximation error for physics-informed (and) operator learningcited by 65
  29. 2015Operator-valued Kernels for Learning from Functional Response Datacited by 59
  30. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  31. 2021Global-Local Transformer for Brain Age Estimationcited by 34
  32. 2020Numerical Solution of Inverse Problems by Weak Adversarial Networkscited by 29
  33. 2020Solving inverse-PDE problems with physics-aware neural networkscited by 27
  34. 2014Functional Regressioncited by 26
  35. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  36. 2017Deep D-bar: Real time Electrical Impedance Tomography Imaging with Deep Neural Networkscited by 24
  37. 2018BCR-Net: a neural network based on the nonstandard wavelet formcited by 18
  38. 2019Solving Electrical Impedance Tomography with Deep Learningcited by 18

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 7; 2025: 5; 2026: 1.

22 papers.

Sort:

  1. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  2. 2022Mitigating spectral bias for the multiscale operator learningcited by 43
  3. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  4. 2023ViTO: Vision Transformer-Operatorcited by 29
  5. 2023Physics Informed Token Transformer for Solving Partial Differential Equationscited by 21
  6. 2023Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)cited by 16
  7. 2023Inducing Point Operator Transformer: A Flexible and Scalable Architecture for Solving PDEscited by 16
  8. 2023Multi-scale Time-stepping of Partial Differential Equations with Transformerscited by 14
  9. 2024Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equationscited by 9
  10. 2025Transformer learns the cross-task prior and regularization for in-context learningcited by 7
  11. 2024CaFA: Global Weather Forecasting with Factorized Attention on Spherecited by 6
  12. 2024Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equationscited by 5
10 more
  1. 2025Learning Causal Graphs at Scale: A Foundation Model Approachcited by 4
  2. 2023Volume-Preserving Transformers for Learning Time Series Data with Structurecited by 3
  3. 2024Green Multigrid Networkcited by 3
  4. 2025Orthogonal greedy algorithm for linear operator learning with shallow neural networkcited by 3
  5. 2023Electrical Impedance Tomography: A Fair Comparative Study on Deep Learning and Analytic-based Approachescited by 2
  6. 2024Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equationscited by 2
  7. 2024Universal Approximation of Operators with Transformers and Neural Integral Operatorscited by 2
  8. 2026In Transformer We Trust? A Perspective on Transformer Architecture Failure Modescited by 1
  9. 2025Mollifier Layers: Enabling Efficient High-Order Derivatives in Inverse PDE Learningcited by 0
  10. 2025A Warm-basis Method for Bridging Learning and Iteration: a Case Study in Fluorescence Molecular Tomographycited by 0