mlsanityData to 7 Oct 2026

Learning Universal Predictors

arXiv 2401.14953 ↗First posted 26 Jan 2024cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 25 of its 54 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

25 of 54 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  9. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  10. 2022Transformers Learn Shortcuts to Automatacited by 260
  11. 2022Neural Networks and the Chomsky Hierarchycited by 214
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
13 more
  1. 2021Pre-training without Natural Imagescited by 100
  2. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  3. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  4. 2019Meta-learning of Sequential Strategiescited by 68
  5. 2017Smart Augmentation - Learning an Optimal Data Augmentation Strategycited by 54
  6. 2020Meta-trained agents implement Bayes-optimal agentscited by 37
  7. 2019Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languagescited by 31
  8. 2017Recurrent Neural Networks as Weighted Language Recognizerscited by 30
  9. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  10. 2023Memory-Based Meta-Learning on Non-Stationary Distributionscited by 14
  11. 2023On the Computational Complexity and Formal Hierarchy of Second Order Recurrent Neural Networkscited by 8
  12. 2020A provably stable neural network Turing Machinecited by 7
  13. 2016Loss Bounds and Time Complexity for Speed Priorscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 9; 2026: 10.

29 papers.

Sort:

By area:

  1. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  2. 2024Lossless data compression by large modelscited by 20
  3. 2024Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Datacited by 13
  4. 2024Ask, and it shall be given: On the Turing completeness of promptingcited by 10
  5. 2024Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Needcited by 10
  6. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  7. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  8. 2025AI Agents as Universal Task Solverscited by 9
  9. 2024In-context learning and Occam's razorcited by 7
  10. 2024Transformers As Approximations of Solomonoff Inductioncited by 6
  11. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  12. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
17 more
  1. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  2. 2025Universal pre-training by iterated random computationcited by 5
  3. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  4. 2024Neural Networks Generalize on Low Complexity Datacited by 3
  5. 2024Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Promptscited by 2
  6. 2024RL, but don't do anything I wouldn't docited by 2
  7. 2026Solomonoff inductioncited by 2
  8. 2026Neural Weight Norm = Kolmogorov Complexitycited by 2
  9. 2026Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesiscited by 1
  10. 2026Position: Universal Time Series Foundation Models Rest on a Category Errorcited by 1
  11. 2026What does a Bayes-filtered transformer believe? A predictive Monte Carlo approachcited by 1
  12. 2025Learning Model Successorscited by 0
  13. 2025Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspectivecited by 0
  14. 2026Can In-Context Learning Support Intrinsic Curiosity?cited by 0
  15. 2026Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posteriorcited by 0
  16. 2026Hierarchical Solomonoff Induction: An Unbounded Machine Learning Modelcited by 0
  17. 2026Understanding Generalization Requires Universal Inductioncited by 0