# LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

arXiv: https://arxiv.org/abs/2402.00795 · HTML: https://mlsanity.com/paper/2402.00795/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputation](https://mlsanity.com/paper/2311.14332/) (2023)
- [Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament](https://mlsanity.com/paper/2310.13014/) (2023)
- [GraB: Finding Provably Better Data Permutations than Random Reshuffling](https://mlsanity.com/paper/2205.10733/) (2022)
- [Transformer Multivariate Forecasting: Less is More?](https://mlsanity.com/paper/2401.00230/) (2023)

## Built on by (most cited first, 30 of 32)

- [Large Language Models as Markov Chains](https://mlsanity.com/paper/2410.02724/) (2024)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [Enough Coin Flips Can Make LLMs Act Bayesian](https://mlsanity.com/paper/2503.04722/) (2025)
- [Bayesian scaling laws for in-context learning](https://mlsanity.com/paper/2410.16531/) (2024)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time](https://mlsanity.com/paper/2510.01098/) (2025)
- [The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors](https://mlsanity.com/paper/2602.02315/) (2026)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [Pretrain-Test Task Alignment Governs Generalization in In-Context Learning](https://mlsanity.com/paper/2509.26551/) (2025)
- [Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery](https://mlsanity.com/paper/2505.16477/) (2025)
- [Large Language Models and Emergence: A Complex Systems Perspective](https://mlsanity.com/paper/2506.11135/) (2025)
- [Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanism](https://mlsanity.com/paper/2509.06322/) (2025)
- [Density estimation with LLMs: a geometric investigation of in-context learning trajectories](https://mlsanity.com/paper/2410.05218/) (2024)
- [Zero-shot Model-based Reinforcement Learning using Large Language Models](https://mlsanity.com/paper/2410.11711/) (2024)
- [Agentic LLM Framework for Adaptive Decision Discourse](https://mlsanity.com/paper/2502.10978/) (2025)
- [Lines of Thought in Large Language Models](https://mlsanity.com/paper/2410.01545/) (2024)
- [How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillator](https://mlsanity.com/paper/2405.17209/) (2024)
- [Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds](https://mlsanity.com/paper/2502.02869/) (2025)
- [Pre-trained Large Language Models Learn Hidden Markov Models In-context](https://mlsanity.com/paper/2506.07298/) (2025)
- [Jacobian Scopes: token-level causal attributions in LLMs](https://mlsanity.com/paper/2601.16407/) (2026)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness](https://mlsanity.com/paper/2607.08046/) (2026)
- [Exploring the Learning Capabilities of Language Models using LEVERWORLDS](https://mlsanity.com/paper/2410.00519/) (2024)
- [In-Context Learning can Perform Continual Learning Like Humans](https://mlsanity.com/paper/2509.22764/) (2025)
- [Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch](https://mlsanity.com/paper/2607.03660/) (2026)
- [Large Language Model Predicts Above Normal All India Summer Monsoon Rainfall in 2024](https://mlsanity.com/paper/2409.16799/) (2024)
- [TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations](https://mlsanity.com/paper/2507.19747/) (2025)
- [Deficiency of equation-finding approach to data-driven modeling of dynamical systems](https://mlsanity.com/paper/2509.03769/) (2025)

## Abstract (arXiv)

Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.