When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference
Not on the map. Why?
Abstract (arXiv)
Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or harms. We benchmark one fixed hand-crafted knowledge source, a pinned bank of Gabor targets injected only during training at ~2% overhead, against data-driven alternatives (SimCLR, SimSiam, DINO, ImageNet transfer, augmentation, learned teachers) under one frozen recipe with fixed subsets: 13 datasets, 9 backbones, 150 to 1.28M images, 32--224,px, 2.5M--86M parameters (computeCells classification configurations over computeRuns runs, plus segmentation and detection transplants). Across the training-time combinations we measure, three outcomes recur (decision-level fusion differs). Different-currency sources can stack: the prior composes with DeiT augmentation on attention backbones and is worth +26 points to ViT-B/16 at 224,px, +6.7 at twice that budget. Same-currency sources substitute: against effective self-supervised pretraining, the combination never usefully exceeds the better single source. Fusing at full strength into an already-informed initialization interferes in proportion to what it carries: ImageNet transfer, -15 to -17 points, removed by a weaker auxiliary weight. Frozen-feature diagnostics measured on each source alone separate these outcomes retrospectively but do not predict them: a rule built on them calls one of nine unseen pairs. At a practitioner's own label budget, the frozen-feature gain predicts the end-to-end gain to within 0.17 points across 30 cells and seven datasets; the underlying decomposition, Delta = G + readout(base), holds in sign on auditRate% of testable cells and is called an unseen backbone family's feature gain in advance. The project page is https://amughrabi.github.io/MomentAux.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
31 of 53 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2022A ConvNet for the 2020scited by 3,545
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2019Searching for MobileNetV3cited by 1,984
- 2014FitNets: Hints for Thin Deep Netscited by 1,619
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
19 more
- 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
- 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
- 2018Rethinking ImageNet Pre-trainingcited by 376
- 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
- 2014Deeply-Supervised Netscited by 364
- 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2018Characterizing and Avoiding Negative Transfercited by 226
- 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 203
- 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
- 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
- 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
- 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
- 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
- 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
- 2017Gabor Convolutional Networkscited by 55
- 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
- 2021Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Taskscited by 15
- 2024FViT: A Focal Vision Transformer with Gabor Filtercited by 3
Built on by
Not cited yet.