mlsanityData to 14 Oct 2026

The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era

arXiv 2608.28980 ↗First posted 29 Aug 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Specialized machine learning architectures encode structural assumptions (equivariance, permutation invariance, relational structure) that language-based foundation models lack by design. This review asks whether such assumptions can instead be acquired through language, drawing on a corpus of 186 papers published between 2016 and 2026 across nine modalities: tabular data, graphs, time series, vision, chemistry, code, knowledge graphs, point clouds, and protein structure. Methods are organized into eight representational regimes, ranging from language-only prompting to fully specialized architectures, and are assessed not only on predictive accuracy but on whether structural information is preserved by the representation and computed by the model. Language-mediated models prove competitive in extreme few-shot prediction, discretized symbolic tasks, and textually annotated knowledge graphs. No evidence of general replacement survives once structure itself is evaluated: where direct tests exist, apparent parity is traceable to information asymmetry, benchmark contamination, or computation performed outside the language model. Across independent research communities, missing structural inductive biases are reintroduced through graph modules, structure-aware tokens, or specialized attention, indicating that specialization is relocated rather than eliminated. The review closes by specifying a falsifiable controlled experiment that would measure the remaining gap directly. The derived data supporting the findings of this review are openly available in the repository at https://github.com/kiyan-rezaee/language-vs-structure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 84 of its 111 references that are in the map. First posted on arXiv 29 Aug 2026.

Builds on

84 of 111 references.

References · 84

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  2. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  3. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  5. 2016Group Equivariant Convolutional Networkscited by 1,165
  6. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  7. 2022A Generalist Agentcited by 808
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
38 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  2. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  5. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  6. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  7. 2023Augmented Language Models: a Surveycited by 403
  8. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  11. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  12. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  13. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 226
  14. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  15. 2024LLaGA: Large Language and Graph Assistantcited by 181
  16. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  17. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  18. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  19. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  20. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  21. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 120
  22. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  23. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  24. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 108
  25. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  26. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  27. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 91
  28. 2024Position: Graph Foundation Models are Already Herecited by 86
  29. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 76
  30. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  31. 2024A Survey of Large Language Models for Graphscited by 72
  32. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinementcited by 58
  33. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
  34. 2025Graph Foundation Models: A Comprehensive Surveycited by 44
  35. 2023Exploring Large Language Models for Knowledge Graph Completioncited by 43
  36. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 38
  37. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  38. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 36

50 most cited of 84 All 84 (JSON)

Built on by

Not cited yet.