mlsanityData to 14 Oct 2026

Context-level Language Modeling by Learning Predictive Context Embeddings

arXiv 2510.20280 ↗First posted 23 Oct 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

35 of 51 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
23 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  13. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  14. 2024The pitfalls of next-token predictioncited by 160
  15. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  16. 2015Character-Aware Neural Language Modelscited by 132
  17. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
  18. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  19. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  20. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  21. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  22. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  23. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  2. 2026Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Modelscited by 11
  3. 2026Full-bandwidth transformercited by 5
  4. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 3
  5. 2026From Interface to Inference: Eliciting Any-Order Inference from Any-Order Modelscited by 1
  6. 2026NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Predictioncited by 0