# Causal Foundation Models: Disentangling Physics from Instrument Properties

arXiv: https://arxiv.org/abs/2507.05333 · HTML: https://mlsanity.com/paper/2507.05333/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 6 of 6)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [AstroM3: A self-supervised multimodal model for astronomy](https://mlsanity.com/paper/2411.08842/) (2024)
- [Maven: A Multimodal Foundation Model for Supernova Science](https://mlsanity.com/paper/2408.16829/) (2024)

## Built on by (most cited first, 3 of 3)

- [Simulation-Based Pretraining and Domain Adaptation for Astronomical Time Series with Minimal Labeled Data](https://mlsanity.com/paper/2510.12958/) (2025)
- [Causal Discovery Should Embrace the Wisdom of the Crowd](https://mlsanity.com/paper/2603.02678/) (2026)
- [Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics](https://mlsanity.com/paper/2604.09787/) (2026)

## Abstract (arXiv)

Foundation models for structured time series data must contend with a fundamental challenge: observations often conflate the true underlying physical phenomena with systematic distortions introduced by measurement instruments. This entanglement limits model generalization, especially in heterogeneous or multi-instrument settings. We present a causally-motivated foundation model that explicitly disentangles physical and instrumental factors using a dual-encoder architecture trained with structured contrastive learning. Leveraging naturally occurring observational triplets (i.e., where the same target is measured under varying conditions, and distinct targets are measured under shared conditions) our model learns separate latent representations for the underlying physical signal and instrument effects. Evaluated on simulated astronomical time series designed to resemble the complexity of variable stars observed by missions like NASA's Transiting Exoplanet Survey Satellite (TESS), our method significantly outperforms traditional single-latent space foundation models on downstream prediction tasks, particularly in low-data regimes. These results demonstrate that our model supports key capabilities of foundation models, including few-shot generalization and efficient adaptation, and highlight the importance of encoding causal structure into representation learning for structured data.