Feature Selection for Tabular Foundation Models: Exploring Feature Utility in Low-Data Molecular Prediction

Choosing features is difficult with few labelled molecules, even for a tabular foundation model that learns in context. We study this problem with TabPFN for low-data molecular property prediction, separating input width, feature meaning and predictive usefulness while keeping contexts, query molecules and predictor settings matched. On ADME/Tox assays from Therapeutics Data Commons, endpoint-matched assay predictions improve a compact RDKit descriptor baseline. A provenance audit finds that most audited query molecules already occur in the predictors' training sources. Across 77 FS-Mol tasks drawn from ChEMBL, broad RDKit descriptor and fingerprint families provide a strong default, and learned weighting of their predictions adds further gains at a larger prediction budget. A utility-trained selector builds a traceable 16-column input from 238 source-assay predictors. It modestly improves the baseline but still trails wider representations. Named outputs identify what each feature predicts and where it comes from. Their usefulness for a new assay is assessed alongside the other input features. Paired removal and replacement tests measure what each selected output contributes. Together, these studies offer a protocol for evaluating feature selection, representation choice and data provenance with a fixed tabular foundation model for few-shot molecular property prediction.