mlsanityData to 7 Oct 2026

Arrows of Time for Large Language Models

arXiv 2401.17505 ↗First posted 30 Jan 2024cs.LG cs.AI cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951). For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

35 of 58 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
23 more
  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  2. 2017Non-Autoregressive Neural Machine Translationcited by 457
  3. 2015Order Matters: Sequence to sequence for setscited by 236
  4. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  5. 2021Towards Causal Representation Learningcited by 207
  6. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  7. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  8. 2019Insertion Transformer: Flexible Sequence Generation via Insertion Operationscited by 123
  9. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  10. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  11. 2023Bayesian Flow Networkscited by 62
  12. 2019Insertion-based Decoding with automatically Inferred Generation Ordercited by 43
  13. 2018Regularizing Neural Machine Translation by Target-bidirectional Agreementcited by 40
  14. 2023Meet in the Middle: A New Pre-training Paradigmcited by 22
  15. 2019KERMIT: Generative Insertion-Based Modeling for Sequencescited by 21
  16. 2018Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Mattercited by 18
  17. 2019A Generalized Framework of Sequence Generation with Application to Undirected Sequence Modelscited by 17
  18. 2015Backward and Forward Language Modeling for Constrained Sentence Generationcited by 9
  19. 2018The Importance of Generation Order in Language Modelingcited by 9
  20. 2019Sequence Modeling with Unconstrained Generation Ordercited by 8
  21. 2019LSTM vs. GRU vs. Bidirectional RNN for script generationcited by 7
  22. 2023Boolformer: Symbolic Regression of Logic Functions with Transformerscited by 5
  23. 2019An Empirical Study of Generation Order for Machine Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  2. 2024The pitfalls of next-token predictioncited by 156
  3. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  4. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  5. 2026Language Modeling with Hyperspherical Flowscited by 15
  6. 2024The Belief State Transformercited by 12
  7. 2025Improving Diversity in Language Models: When Temperature Fails, Change the Losscited by 7
  8. 2025Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measurescited by 2
  9. 2024Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Textcited by 1
  10. 2025What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?cited by 1
  11. 2025Distilling Specialized Orders for Visual Generationcited by 1
  12. 2025Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepanciescited by 1
6 more
  1. 2025Directional Optimization Asymmetry in Transformers: A Synthetic Stress Testcited by 0
  2. 2026Marginals Before Conditionalscited by 0
  3. 2026BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplerscited by 0
  4. 2026Simplifying the Modeling of Arbitrary Conditionals in Natural Languagecited by 0
  5. 2026A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Textscited by 0
  6. 2026Line-Coupled Language Modelcited by 0