mlsanityData to 14 Oct 2026

Memory-Based Meta-Learning on Non-Stationary Distributions

arXiv 2302.03067 ↗First posted 6 Feb 2023cs.LG cs.AI stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Memory-based meta-learning is a technique for approximating Bayes-optimal predictors. Under fairly general conditions, minimizing sequential prediction error, measured by the log loss, leads to implicit meta-learning. The goal of this work is to investigate how far this interpretation can be realized by current sequence prediction models and training regimes. The focus is on piecewise stationary sources with unobserved switching-points, which arguably capture an important characteristic of natural language and action-observation sequences in partially observable environments. We show that various types of memory-based neural models, including Transformers, LSTMs, and RNNs can learn to accurately approximate known Bayes-optimal algorithms and behave as if performing Bayesian inference over the latent switching-points and the latent parameters governing the data distribution within each segment.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

16 of 30 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022A Generalist Agentcited by 808
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  8. 2016Learning to reinforcement learncited by 377
  9. 2021Transformers Can Do Bayesian Inferencecited by 360
  10. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  11. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  12. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
4 more
  1. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  2. 2019Meta-learning of Sequential Strategiescited by 70
  3. 2020Meta-trained agents implement Bayes-optimal agentscited by 38
  4. 2016Deep Amortized Inference for Probabilistic Programscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 3; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Language Modeling Is Compressioncited by 225
  2. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  3. 2024Learning Universal Predictorscited by 31
  4. 2024Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Datacited by 13
  5. 2023Implicit meta-learning may lead language models to trust more reliable sourcescited by 10
  6. 2024In-context learning and Occam's razorcited by 8
  7. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
  8. 2024Transformers for Supervised Online Continual Learningcited by 5
  9. 2023Meta-Learning Strategies through Value Maximization in Neural Networkscited by 4
  10. 2023Task-Distributionally Robust Data-Free Meta-Learningcited by 4
  11. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  12. 2025Heterogeneous Adversarial Play in Interactive Environmentscited by 3
3 more
  1. 2026What does a Bayes-filtered transformer believe? A predictive Monte Carlo approachcited by 1
  2. 2026Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posteriorcited by 0
  3. 2026Learning Source Structure: Bayesian Prediction with Transformerscited by 0