# Mechanistic Neural Networks for Scientific Machine Learning

arXiv: https://arxiv.org/abs/2402.13077 · HTML: https://mlsanity.com/paper/2402.13077/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization](https://mlsanity.com/paper/1809.05504/) (2018)
- [SINDy-PI: A Robust Algorithm for Parallel Implicit Sparse Identification of Nonlinear Dynamics](https://mlsanity.com/paper/2004.02322/) (2020)
- [Lie Point Symmetry Data Augmentation for Neural PDE Solvers](https://mlsanity.com/paper/2202.07643/) (2022)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [On the Differentiability of the Solution to Convex Optimization Problems](https://mlsanity.com/paper/1804.05098/) (2018)
- [ODIN: ODE-Informed Regression for Parameter and State Inference in Time-Continuous Dynamical Systems](https://mlsanity.com/paper/1902.06278/) (2019)
- [Differentiable Mathematical Programming for Object-Centric Representation Learning](https://mlsanity.com/paper/2210.02159/) (2022)

## Built on by (most cited first, 16 of 16)

- [Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems](https://mlsanity.com/paper/2504.02918/) (2025)
- [Marrying Causal Representation Learning with Dynamical Systems for Science](https://mlsanity.com/paper/2405.13888/) (2024)
- [Space-Time Continuous PDE Forecasting using Equivariant Neural Fields](https://mlsanity.com/paper/2406.06660/) (2024)
- [Scalable Mechanistic Neural Networks for Differential Equations and Machine Learning](https://mlsanity.com/paper/2410.06074/) (2024)
- [Discovering Physics Laws of Dynamical Systems via Invariant Function Learning](https://mlsanity.com/paper/2502.04495/) (2025)
- [Adaptive Physics-informed Neural Networks: A Survey](https://mlsanity.com/paper/2503.18181/) (2025)
- [Generative Intervention Models for Causal Perturbation Modeling](https://mlsanity.com/paper/2411.14003/) (2024)
- [Mechanistic PDE Networks for Discovery of Governing Equations](https://mlsanity.com/paper/2502.18377/) (2025)
- [Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention](https://mlsanity.com/paper/2603.14483/) (2026)
- [MatrixNet: Learning over symmetry groups using learned group representations](https://mlsanity.com/paper/2501.09571/) (2025)
- [Data-driven Discovery of Digital Twins in Biomedical Research](https://mlsanity.com/paper/2508.21484/) (2025)
- ["Cause" is Mechanistic Narrative within Scientific Domains: An Ordinary Language Philosophical Critique of "Causal Machine Learning"](https://mlsanity.com/paper/2501.05844/) (2025)
- [Learning Explicit Single-Cell Dynamics Using ODE Representations](https://mlsanity.com/paper/2510.02903/) (2025)
- [Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing](https://mlsanity.com/paper/2602.02157/) (2026)
- [TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series Models](https://mlsanity.com/paper/2601.09776/) (2026)
- [Identifying ODEs from Unstructured Data with Causal Representation Learning](https://mlsanity.com/paper/2609.37083/) (2026)

## Abstract (arXiv)

This paper presents Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations, revealing the underlying dynamics of data and enhancing interpretability and efficiency in data modeling. Central to our approach is a novel Relaxed Linear Programming Solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs. This integrates well with neural networks and surpasses the limitations of traditional ODE solvers enabling scalable GPU parallel processing. Overall, Mechanistic Neural Networks demonstrate their versatility for scientific machine learning applications, adeptly managing tasks from equation discovery to dynamic systems modeling. We prove their comprehensive capabilities in analyzing and interpreting complex scientific data across various applications, showing significant performance against specialized state-of-the-art methods.