mlsanityData to 7 Oct 2026

Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs

arXiv 2605.27255 ↗First posted 26 May 2026cs.CL cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Long chain-of-thought reasoning has made autoregressive decoding the dominant inference cost of modern large language models. Existing methods target either the input side (latent compression) or the output side (speculative decoding and multi-token prediction, MTP), but the two lines of work have been pursued independently. Moreover, output-side methods must incur an expensive verifier pass to validate the unreliable draft tokens predicted by MTP. To address these issues, we propose Pair-In, Pair-Out (PIPO), which unifies both sides by viewing a latent compressor and an MTP head as mirror-image operations: the compressor folds two input tokens into one latent representation, while the MTP head unfolds one hidden state into one additional output token. To remove the verifier cost without sacrificing reliability, PIPO trains a lightweight confidence head that decides whether draft tokens should be accepted. We observe that On-Policy Distillation (OPD) naturally matches the rejection-sampling criterion of speculative decoding, so the confidence head can be trained alongside OPD with negligible extra cost. Experiments on AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 with Qwen3.5-4B and 9B backbones show that PIPO improves pass@4 over regular decoding by up to +7.15 points, while delivering up to 2.64× first-token-latency and 2.07× per-token-latency speedups. Project Page: GitHub.com/RedAI-Infra/PIPO.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 20 of its 26 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

20 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains22 May
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan

Other references · 18

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  8. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  10. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 290
  11. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  12. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
6 more
  1. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 109
  2. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 40
  3. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinkingcited by 34
  4. 2026Multiplex Thinking: Reasoning via Token-wise Branch-and-Mergecited by 16
  5. 2025REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understandingcited by 15
  6. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligencecited by 0
  2. 2026Can Computation from Earlier Problems Help LLMs Solve New Ones?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
  2. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
  3. 2025On multi-token prediction for efficient LLM inference
  4. 2026Multi-Token Prediction via Self-Distillation
  5. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  6. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusion
  7. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  8. 2024Tandem Transformers for Inference Efficient LLMs