mlsanityData to 7 Oct 2026

Rethinking Neural Operations for Diverse Tasks

arXiv 2103.15798 ↗First posted 29 Mar 2021cs.LG cs.AI cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users to discover the right neural operations given data from their specific domain. We introduce a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: we prove that it includes many named operations across multiple application areas. Starting with any standard backbone such as ResNet, we show how to transform it into a search space over XD-operations and how to traverse the space using a simple weight-sharing scheme. On a diverse set of tasks -- solving PDEs, distance prediction for protein folding, and music modeling -- our approach consistently yields models with lower error than baseline networks and often even lower error than expert-designed domain-specific approaches.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

39 of 68 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Wide Residual Networkscited by 3,057
  10. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
27 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  2. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  3. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  4. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  5. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  6. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  7. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  8. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  9. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  10. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  11. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  12. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  13. 2019Neural Machine Translation: A Review and Surveycited by 99
  14. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  15. 2019NAS evaluation is frustratingly hardcited by 69
  16. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  17. 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
  18. 2018Trellis Networks for Sequence Modelingcited by 50
  19. 2019AtomNAS: Fine-Grained End-to-End Neural Architecture Searchcited by 40
  20. 2020Geometry-Aware Gradient Algorithms for Neural Architecture Searchcited by 40
  21. 2018Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cellscited by 39
  22. 2019R-Transformer: Recurrent Neural Network Enhanced Transformercited by 39
  23. 2020Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Mapscited by 36
  24. 2015ACDC: A Structured Efficient Linear Layercited by 19
  25. 2019Butterfly Transform: An Efficient FFT Based Neural Architecture Designcited by 16
  26. 2019TextNAS: A Neural Architecture Search Space tailored for Text Representationcited by 16
  27. 2020Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practicecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 5; 2024: 3; 2025: 1; 2026: 1.

18 papers.

Sort:

By area:

  1. 2021Choose a Transformer: Fourier or Galerkincited by 333
  2. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  3. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  4. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  5. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 23
  6. 2022Efficient Architecture Search for Diverse Taskscited by 18
  7. 2022AANG: Automating Auxiliary Learningcited by 12
  8. 2022Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammarscited by 9
  9. 2022Efficient Non-Parametric Optimizer Search for Diverse Taskscited by 6
  10. 2024einspace: Searching for Neural Architectures from Fundamental Operationscited by 5
  11. 2022AutoML for Climate Change: A Call to Actioncited by 3
  12. 2023Learning Functional Transductioncited by 2
6 more
  1. 2023How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictorcited by 2
  2. 2024Pretrained Hybrids with MAD Skillscited by 2
  3. 2023An Evaluation of Zero-Cost Proxies -- from Neural Architecture Performance to Model Robustnesscited by 1
  4. 2025COSMOS: Predictable and Cost-Effective Adaptation of LLMscited by 1
  5. 2026Agentic Neural Architecture Searchcited by 1
  6. 2022Pseudo-Inverted Bottleneck Convolution for DARTS Search Spacecited by 0