# Transformer-Powered Surrogates Close the ICF Simulation-Experiment Gap with Extremely Limited Data

arXiv: https://arxiv.org/abs/2312.03642 · HTML: https://mlsanity.com/paper/2312.03642/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 56 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Mapping the Space of Chemical Reactions Using Attention-Based Neural Networks](https://mlsanity.com/paper/2012.06051/) (2020)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Regression Transformer: Concurrent sequence regression and generation for molecular language modeling](https://mlsanity.com/paper/2202.01338/) (2022)
- [Few-shot Learning with Noisy Labels](https://mlsanity.com/paper/2204.05494/) (2022)
- [A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias](https://mlsanity.com/paper/2303.13500/) (2023)
- [Transfer learning to model inertial confinement fusion experiments](https://mlsanity.com/paper/1812.06055/) (2018)
- [Improved Surrogates in Inertial Confinement Fusion with Manifold and Cycle Consistencies](https://mlsanity.com/paper/1912.08113/) (2019)
- [RNNP: A Robust Few-Shot Learning Approach](https://mlsanity.com/paper/2011.11067/) (2020)
- [Coverage-Based Designs Improve Sample Mining and Hyper-Parameter Optimization](https://mlsanity.com/paper/1809.01712/) (2018)
- [Suppressing simulation bias using multi-modal data](https://mlsanity.com/paper/2104.09684/) (2021)

## Built on by (most cited first, 2 of 2)

- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [Co4ICF: Co-evolving Physics-Informed Surrogate and RL-based Pulse Optimizer for Inertial Confinement Fusion](https://mlsanity.com/paper/2607.10366/) (2026)

## Abstract (arXiv)

Recent advances in machine learning, specifically transformer architecture, have led to significant advancements in commercial domains. These powerful models have demonstrated superior capability to learn complex relationships and often generalize better to new data and problems. This paper presents a novel transformer-powered approach for enhancing prediction accuracy in multi-modal output scenarios, where sparse experimental data is supplemented with simulation data. The proposed approach integrates transformer-based architecture with a novel graph-based hyper-parameter optimization technique. The resulting system not only effectively reduces simulation bias, but also achieves superior prediction accuracy compared to the prior method. We demonstrate the efficacy of our approach on inertial confinement fusion experiments, where only 10 shots of real-world data are available, as well as synthetic versions of these experiments.