mlsanityData to 14 Oct 2026

PonderLM: Pretraining Language Models to Ponder in Continuous Space

arXiv 2505.20674 ↗First posted 27 May 2025cs.CL cs.AIcited by 44

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort. In this work, we introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step. During pondering, instead of generating an actual token sampled from the prediction distribution, the model ponders by yielding a weighted sum of all token embeddings according to the predicted token distribution. The generated embedding is then fed back as input for another forward pass. We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations. Experiments across three widely used open-source architectures-GPT-2, Pythia, and LLaMA-and extensive downstream task evaluations demonstrate the effectiveness and generality of our method. On 9 downstream benchmarks, our pondering-enhanced Pythia models significantly outperform the official Pythia models. Notably, our PonderPythia models demonstrate remarkable effectiveness: PonderPythia-2.8B surpasses Pythia-6.9B and rivals Pythia-12B, while our PonderPythia-1B matches TinyLlama-1.1B, a model trained on 10 times more data. The code is available at https://github.com/LUMIA-Group/PonderingLM.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 65 of its 81 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

65 of 81 references.

References · 65

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2019Root Mean Square Layer Normalizationcited by 1,480
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  14. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  15. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  16. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  17. 2022Large Language Models Can Self-Improvecited by 664
  18. 2018Universal Transformerscited by 647
  19. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  20. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  21. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  22. 2023Scaling Data-Constrained Language Modelscited by 403
  23. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  24. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  25. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  26. 2023Looped Transformers as Programmable Computerscited by 237
  27. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 191
  28. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  29. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  30. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  31. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  32. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  33. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  34. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  35. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  36. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  37. 2022Block-Recurrent Transformerscited by 110
  38. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 32.

44 papers.

Sort:

By area:

  1. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  3. 2025A Survey on Latent Reasoningcited by 81
  4. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  5. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  6. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  7. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  8. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 18
  9. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  10. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 16
  11. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 15
  12. 2026LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Modelscited by 15
32 more
  1. 2026Training-Free Looped Transformerscited by 15
  2. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  3. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  4. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 10
  5. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  6. 2026Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs?cited by 8
  7. 2025Context-level Language Modeling by Learning Predictive Context Embeddingscited by 6
  8. 2026The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoningcited by 6
  9. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 5
  10. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 5
  11. 2026Capabilities and Fundamental Limits of Latent Chain-of-Thoughtcited by 4
  12. 2026PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Maskingcited by 4
  13. 2026DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoningcited by 4
  14. 2026The Topological Trouble With Transformerscited by 3
  15. 2026Depth-Attention: Cross-Layer Value Mixing for Language Modelscited by 3
  16. 2026LoopMTP: A looped transformer guided by latent multi-token predictioncited by 3
  17. 2025Catch Your Breath: Adaptive Computation for Self-Paced Sequence Productioncited by 2
  18. 2026Training Continuous Chain of Thought Models: A Tale of Two Regimescited by 2
  19. 2026Turbo Connection: Reasoning as Information Flow from Higher to Lower Layerscited by 1
  20. 2026Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoningcited by 1
  21. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  22. 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
  23. 2026A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLMcited by 1
  24. 2026Quantizing Looped Transformers: Feedback Exposure and Calibration Blindnesscited by 1
  25. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1
  26. 2026A First-Principles Theory of Slow Thinking and Active Perceptioncited by 0
  27. 2026HALO: Hybrid Adaptive Latent Reasoning for Language Modelscited by 0
  28. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  29. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0
  30. 2026Write Back the Delta: Revisiting the Same Tokens with Fresh Representationscited by 0
  31. 2026Pretraining Latent Information Feedback Transformers with Teacher Supervisioncited by 0
  32. 2026SkillFM: Generating Skills for LLM Agents via Latent Flow Matchingcited by 0