# Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

arXiv: https://arxiv.org/abs/2603.04692 · HTML: https://mlsanity.com/paper/2603.04692/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [Benchmarking the Performance of Bayesian Optimization across Multiple Experimental Materials Science Domains](https://mlsanity.com/paper/2106.01309/) (2021)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [DrivAerNet: A Parametric Car Dataset for Data-Driven Aerodynamic Design and Prediction](https://mlsanity.com/paper/2403.08055/) (2024)
- [FRAMED: An AutoML Approach for Structural Performance Prediction of Bicycle Frames](https://mlsanity.com/paper/2201.10459/) (2022)
- [t-METASET: Tailoring Property Bias of Large-Scale Metamaterial Datasets through Active Learning](https://mlsanity.com/paper/2202.10565/) (2022)
- [IM-Context: In-Context Learning for Imbalanced Regression Tasks](https://mlsanity.com/paper/2405.18202/) (2024)
- [Model-Free Data-Driven Inference in Computational Mechanics](https://mlsanity.com/paper/2207.06419/) (2022)
- [Simultaneous and Meshfree Topology Optimization with Physics-informed Gaussian Processes](https://mlsanity.com/paper/2408.03490/) (2024)
- [BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints](https://mlsanity.com/paper/2508.00830/) (2025)
- [Interpretable Multi-Source Data Fusion Through Latent Variable Gaussian Process](https://mlsanity.com/paper/2402.04146/) (2024)
- [GUIDe: Generative and Uncertainty-Informed Inverse Design for On-Demand Nonlinear Functional Responses](https://mlsanity.com/paper/2509.05641/) (2025)
- [A Data-driven Recommendation Framework for Optimal Walker Designs](https://mlsanity.com/paper/2310.18772/) (2023)

## Built on by (most cited first, 3 of 3)

- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [A Framework and Prototype for a Navigable Map of Datasets in Engineering Design and Systems Engineering](https://mlsanity.com/paper/2603.15722/) (2026)
- [TabMF: Learning Dataset Representations with Tabular Foundation Models](https://mlsanity.com/paper/2610.13818/) (2026)

## Abstract (arXiv)

Predictive modeling in engineering applications has long been dominated by bespoke models and small, siloed tabular datasets, limiting the applicability of large-scale learning approaches. Despite recent progress in tabular foundation models, the resulting synthetic training distributions used for pre-training may not reflect the statistical structure of engineering data, limiting transfer to engineering regression. We introduce TREDBench, a curated collection of 83 real-world tabular regression datasets with expert engineering/non-engineering labels, and use TabPFN 2.5's dataset-level embedding to study domain structure in a common representation space. We find that engineering datasets are partially distinguishable from non-engineering datasets, while standard procedurally generated datasets are highly distinguishable from engineering datasets, revealing a substantial synthetic-real domain gap. To bridge this gap without training on real engineering samples, we propose an embedding-guided synthetic data curation method: we generate and identify "engineering-like" synthetic datasets, and perform continued pre-training of TabPFN 2.5 using only the selected synthetic tasks. Across 35 engineering regression datasets, this synthetic-only adaptation improves predictive accuracy and data efficiency, outperforming TabPFN 2.5 on 29/35 datasets and AutoGluon on 27/35, with mean multiplicative data-efficiency gains of 1.75x and 4.44x, respectively. More broadly, our results indicate that principled synthetic data curation can convert procedural generators into domain-relevant "data engines," enabling foundation models to improve in data-sparse scientific and industrial domains where real data collection is the primary bottleneck.