mlsanityData to 14 Oct 2026

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

arXiv 2605.06510 ↗First posted 7 May 2026cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

43 of 53 references.

References · 43

  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  7. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  8. 2018Universal Transformerscited by 647
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  12. 2021Transformers Can Do Bayesian Inferencecited by 360
31 more
  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  3. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  4. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  5. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
  6. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  7. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  8. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  9. 2024Universal Neurons in GPT2 Language Modelscited by 102
  10. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  11. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  12. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  13. 2025LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligencecited by 79
  14. 2017Residual Connections Encourage Iterative Inferencecited by 77
  15. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 76
  16. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  17. 2023Statistical Foundations of Prior-Data Fitted Networkscited by 65
  18. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  19. 2024Transformer Layers as Painterscited by 54
  20. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  21. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  22. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  23. 2024Explorations of Self-Repair in Language Modelscited by 39
  24. 2025nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFNcited by 15
  25. 2025Early Stopping Tabular In-Context Learningcited by 12
  26. 2024PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applicationscited by 6
  27. 2025LLM Interpretability with Identifiable Temporal-Instantaneous Representationcited by 6
  28. 2025What exactly has TabPFN learned to do?cited by 5
  29. 2025From Tables to Signals: Revealing Spectral Adaptivity in TabPFNcited by 4
  30. 2025Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architecturescited by 3
  31. 2025What Makes Looped Transformers Perform Better Than Non-Recursive Onescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026TACTICL: Task-Aware Compression of Tabular ICL Modelscited by 4
  2. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  3. 2026Latent Chain-of-Thought Improves Structured-Data Transformerscited by 0
  4. 2026LoopICL: Looping a single transformer block to solve tabular taskscited by 0
  5. 2026Architecture Alignment With Sparse Priors in Tabular Foundation Modelscited by 0
  6. 2026TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversitycited by 0
  7. 2026GeneICL: A Tabular Foundation Model for Bulk Transcriptomicscited by 0
  8. 2026Tabular Foundation Models Are Effectively Shallowcited by 0
  9. 2026Recurrent In-Context Learning for Tabular Foundation Modelscited by 0
  10. 2026When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Modelscited by 0