# Self-Conditioned Denoising for Atomistic Representation Learning

arXiv: https://arxiv.org/abs/2603.17196 · HTML: https://mlsanity.com/paper/2603.17196/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems](https://mlsanity.com/paper/2307.08423/) (2023)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [All-atom Diffusion Transformers: Unified generative modelling of molecules and materials](https://mlsanity.com/paper/2503.03965/) (2025)
- [Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching](https://mlsanity.com/paper/2206.13602/) (2022)
- [Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?](https://mlsanity.com/paper/2201.05119/) (2022)
- [Learning Hierarchical Protein Representations via Complete 3D Graph Networks](https://mlsanity.com/paper/2207.12600/) (2022)
- [Multi-Scale Representation Learning on Proteins](https://mlsanity.com/paper/2204.02337/) (2022)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Sliced Denoising: A Physics-Informed Molecular Pre-Training Method](https://mlsanity.com/paper/2311.02124/) (2023)
- [Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fields](https://mlsanity.com/paper/2403.09549/) (2024)
- [UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning](https://mlsanity.com/paper/2405.10343/) (2024)
- [Pre-training with Fractional Denoising to Enhance Molecular Property Prediction](https://mlsanity.com/paper/2407.11086/) (2024)
- [May the Force be with You: Unified Force-Centric Pre-Training for 3D Molecular Conformations](https://mlsanity.com/paper/2308.14759/) (2023)
- [ProFSA: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment](https://mlsanity.com/paper/2310.07229/) (2023)
- [Geometric Representation Condition Improves Equivariant Molecule Generation](https://mlsanity.com/paper/2410.03655/) (2024)
- [Leveraging neural network interatomic potentials for a foundation model of chemistry](https://mlsanity.com/paper/2506.18497/) (2025)

## Built on by (most cited first, 4 of 4)

- [Harnessing AtomisticSkills for Agentic Atomistic Research](https://mlsanity.com/paper/2605.24002/) (2026)
- [Toward Better Geometric Representations for Molecule Generative Models](https://mlsanity.com/paper/2605.07693/) (2026)
- [Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems](https://mlsanity.com/paper/2610.08400/) (2026)
- [Zatom-2: Multitask Pretraining on Atomistic Data for Generative Modeling across Domains](https://mlsanity.com/paper/2610.11454/) (2026)

## Abstract (arXiv)

The success of large-scale pretraining in NLP and computer vision has catalyzed growing efforts to develop analogous foundation models for the physical sciences. However, pretraining strategies using atomistic data remain underexplored. To date, large-scale supervised pretraining on DFT force-energy labels has provided the strongest performance gains to downstream property prediction, out-performing existing methods of self-supervised learning (SSL) which remain limited to ground-state geometries, and/or single domains of atomistic data. We address these shortcomings with Self-Conditioned Denoising (SCD), a backbone-agnostic reconstruction objective that utilizes self-embeddings for conditional denoising across any domain of atomistic data, including small molecules, proteins, periodic materials, and 'non-equilibrium' geometries. When controlled for backbone architecture and pretraining dataset, SCD significantly outperforms previous SSL methods on downstream benchmarks and matches or exceeds the performance of supervised force-energy pretraining. We show that a small, fast GNN pretrained by SCD can achieve competitive or superior performance to larger models pretrained on significantly larger labeled or unlabeled datasets, across tasks in multiple domains. Our code is available at: https://github.com/TyJPerez/SelfConditionedDenoisingAtoms