mlsanityData to 14 Oct 2026

LLM Pretraining with Continuous Concepts

arXiv 2502.08524 ↗First posted 12 Feb 2025cs.LG cs.CLcited by 36

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 44 of its 57 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

44 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Think before you speak: Training Language Models With Pause Tokens3 Oct

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
31 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2024Scaling and evaluating sparse autoencoderscited by 553
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  13. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  14. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  15. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  16. 2013k-Sparse Autoencoderscited by 215
  17. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  18. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  19. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  20. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  21. 2024The pitfalls of next-token predictioncited by 160
  22. 2024Distilling System 2 into System 1cited by 159
  23. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  24. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  25. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  26. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  27. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  28. 2023Learning to Reason and Memorize with Self-Notescited by 30
  29. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  30. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  31. 2024Semiparametric Token-Sequence Co-Supervisioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 15.

36 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  3. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 68
  4. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  5. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  6. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  7. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  8. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 32
  9. 2025Parallel Loop Transformer for Efficient Test-Time Computation Scalingcited by 24
  10. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  11. 2025Soft-Masked Diffusion Language Modelscited by 16
  12. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12
24 more
  1. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  2. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 10
  3. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  4. 2025Efficient Pretraining Length Scalingcited by 6
  5. 2026Capabilities and Fundamental Limits of Latent Chain-of-Thoughtcited by 4
  6. 2026LEPO: Latent Reasoning Policy Optimization for Large Language Modelscited by 4
  7. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 4
  8. 2025FOL-Traces: Verified First-Order Logic Reasoning Traces at Scalecited by 3
  9. 2025Towards Distributed Neural Architecturescited by 2
  10. 2025Improving Latent Reasoning in LLMs via Soft Concept Mixingcited by 2
  11. 2025iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planningcited by 2
  12. 2026TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RLcited by 2
  13. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 2
  14. 2025Latent Reasoning via Sentence Embedding Predictioncited by 1
  15. 2026Overton Pluralistic Reinforcement Learning for Large Language Modelscited by 1
  16. 2026CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoningcited by 1
  17. 2025VOX-KRIKRI: Unifying Speech and Language through Continuous Fusioncited by 0
  18. 2025Vocabulary In-Context Learning in Transformers: Benefits of Positional Encodingcited by 0
  19. 2025Multi-Path Collaborative Reasoning via Reinforcement Learningcited by 0
  20. 2026LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAGcited by 0
  21. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0
  22. 2026Pretraining Latent Information Feedback Transformers with Teacher Supervisioncited by 0
  23. 2026Hybrid Reasoning Agentscited by 0
  24. 2026INTACT: Inducing Latent Reasoning on a Frozen Backbonecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Tokens: Concept-Level Training Objectives for LLMs
  2. 2026Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
  3. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
  4. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  6. 2026Pretraining with Token-Level Adaptive Latent Chain-of-Thought
  7. 2024Improving Self Consistency in LLMs through Probabilistic Tokenization
  8. 2026DiffuMask: Diffusion Language Model for Token-level Prompt Pruning