mlsanityData to 7 Oct 2026

Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models

arXiv 2602.08984 ↗First posted 9 Feb 2026cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

36 of 48 references.

References · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
24 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  13. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  14. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  15. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  16. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  17. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  18. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  19. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 77
  20. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  21. 2025Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacecited by 14
  22. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  23. 2025Context-level Language Modeling by Learning Predictive Context Embeddingscited by 6
  24. 2025HAMburger: Accelerating LLM Inference via Token Smashingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Full-bandwidth transformercited by 3
  2. 2026VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantizationcited by 1
  3. 2026Efficient Pre-Training with Token Superpositioncited by 1
  4. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 1
  5. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0
  6. 2026From Found to Designed: Concepts as a Design Axis for Large Language Modelscited by 0
  7. 2026A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLMcited by 0
  8. 2026NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Predictioncited by 0