# On Learning what to Learn: heterogeneous observations of dynamics and establishing (possibly causal) relations among them

arXiv: https://arxiv.org/abs/2406.06812 · HTML: https://mlsanity.com/paper/2406.06812/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs](https://mlsanity.com/paper/2003.05425/) (2020)
- [Tackling the Curse of Dimensionality with Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.12306/) (2023)
- [Algorithms for Overcoming the Curse of Dimensionality for Certain Hamilton-Jacobi Equations Arising in Control Theory and Elsewhere](https://mlsanity.com/paper/1605.01799/) (2016)
- [LOCA: LOcal Conformal Autoencoder for standardized data coordinates](https://mlsanity.com/paper/2004.07234/) (2020)
- [Latent common manifold learning with alternating diffusion: analysis and applications](https://mlsanity.com/paper/1602.00078/) (2016)
- [Recovering Hidden Components in Multimodal Data with Composite Diffusion Operators](https://mlsanity.com/paper/1808.07312/) (2018)
- [Spatiotemporal Analysis Using Riemannian Composition of Diffusion Operators](https://mlsanity.com/paper/2201.08530/) (2022)
- [Optimal Recovery of Precision Matrix for Mahalanobis Distance from High Dimensional Noisy Observations in Manifold Learning](https://mlsanity.com/paper/1904.09204/) (2019)
- [Spectral Discovery of Jointly Smooth Features for Multimodal Data](https://mlsanity.com/paper/2004.04386/) (2020)
- [A common variable minimax theorem for graphs](https://mlsanity.com/paper/2107.14747/) (2021)
- [On Geometry of Information Flow for Causal Inference](https://mlsanity.com/paper/2002.02078/) (2020)
- [Spectral Flow on the Manifold of SPD Matrices for Multimodal Data Processing](https://mlsanity.com/paper/2009.08062/) (2020)
- [On the Correspondence between Gaussian Processes and Geometric Harmonics](https://mlsanity.com/paper/2110.02296/) (2021)
- [Gappy local conformal auto-encoders for heterogeneous data fusion: in praise of rigidity](https://mlsanity.com/paper/2312.13155/) (2023)

## Built on by (most cited first, 2 of 2)

- [RANDSMAPs: Random-Feature/multi-Scale Neural Decoders with Mass Preservation](https://mlsanity.com/paper/2601.14794/) (2026)
- [Learning to Choose: An Empowerment-Guided Multi-Agent System with semantic communication for Adaptive Method Selection](https://mlsanity.com/paper/2605.30042/) (2026)

## Abstract (arXiv)

Before we attempt to learn a function between two (sets of) observables of a physical process, we must first decide what the inputs and what the outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of initially deciding ``the right quantities'' to relate through such a function, and then proceed to learn it. This is accomplished by processing multiple simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: multiple observation processes of the system. We thus determine (a) what subsets of observables are common between the observation processes (and therefore observable from each other, relatable through a function); and (b) what information is unrelated to these common observables, and therefore particular to each observation process, and not contributing to the desired function. Any data-driven function approximation technique can subsequently be used to learn the input-output relation, from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular ``twists'' of the approach are discussed. The first has to do with the identifiability of particular quantities of interest from the measurements. We now construct mappings from a single set of observations of one process to entire level sets of measurements of the process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures ``now'', while the second observation process measures ``in the future'', the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution.