mlsanityData to 14 Oct 2026

Hierarchical Latent Prediction for Language Models

arXiv 2608.05806 ↗First posted 6 Aug 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 13 of its 19 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

13 of 19 references.

References · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2016The Option-Critic Architecturecited by 448
  3. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  4. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  5. 2024The pitfalls of next-token predictioncited by 160
  6. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  7. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  8. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  9. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  10. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  11. 2024Learning Discrete Concepts in Latent Hierarchical Modelscited by 17
  12. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 15
1 more
  1. 2025Efficient Joint Prediction of Multiple Future Tokenscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pre-Training Curriculum for Multi-Token Prediction in Language Models
  2. 2026Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate
  3. 2026RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors From Multi-Step to One-Step
  4. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
  5. 2026Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
  6. 2026Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
  7. 2025Efficient Post-Training Refinement of Latent Reasoning in Large Language Models
  8. 2026LatentMT: Machine Translation with Latent Reasoning