mlsanityData to 7 Oct 2026

LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

arXiv 2402.00795 ↗First posted 1 Feb 2024cs.LG cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
15 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  7. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  8. 2023Language Models Represent Space and Timecited by 306
  9. 2023In-Context Learning Creates Task Vectorscited by 302
  10. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  11. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  12. 2023GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputationcited by 35
  13. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
  14. 2022GraB: Finding Provably Better Data Permutations than Random Reshufflingcited by 18
  15. 2023Transformer Multivariate Forecasting: Less is More?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 16; 2026: 7.

32 papers.

Sort:

By area:

  1. 2024Large Language Models as Markov Chainscited by 33
  2. 2024Zero-shot forecasting of chaotic systemscited by 25
  3. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 24
  4. 2025Enough Coin Flips Can Make LLMs Act Bayesiancited by 19
  5. 2024Bayesian scaling laws for in-context learningcited by 18
  6. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  7. 2026The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriorscited by 15
  8. 2025Token embeddings violate the manifold hypothesiscited by 14
  9. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  10. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 12
  11. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  12. 2025Large Language Models and Emergence: A Complex Systems Perspectivecited by 7
20 more
  1. 2025Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanismcited by 7
  2. 2024Density estimation with LLMs: a geometric investigation of in-context learning trajectoriescited by 6
  3. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  4. 2025Agentic LLM Framework for Adaptive Decision Discoursecited by 6
  5. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 6
  6. 2024Lines of Thought in Large Language Modelscited by 5
  7. 2024How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillatorcited by 4
  8. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  9. 2025Pre-trained Large Language Models Learn Hidden Markov Models In-contextcited by 4
  10. 2026Jacobian Scopes: token-level causal attributions in LLMscited by 4
  11. 2026Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASHcited by 2
  12. 2026What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulnesscited by 2
  13. 2024Exploring the Learning Capabilities of Language Models using LEVERWORLDScited by 1
  14. 2025In-Context Learning can Perform Continual Learning Like Humanscited by 1
  15. 2024Large Language Model Predicts Above Normal All India Summer Monsoon Rainfall in 2024cited by 0
  16. 2025TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformationscited by 0
  17. 2025Deficiency of equation-finding approach to data-driven modeling of dynamical systemscited by 0
  18. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 0
  19. 2026A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learningcited by 0
  20. 2026Predicting the Unpredictable: LLM-powered Long-term Chaotic Time Series Forecasting under Short-term Observationscited by 0