mlsanityData to 7 Oct 2026

Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation

arXiv 2307.15337 ↗First posted 28 Jul 2023cs.CL cs.AIcited by 103

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

This work aims at decreasing the end-to-end generation latency of large language models (LLMs). One of the major causes of the high generation latency is the sequential decoding approach adopted by almost all state-of-the-art LLMs. In this work, motivated by the thinking and writing process of humans, we propose Skeleton-of-Thought (SoT), which first guides LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel. Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories. SoT is an initial attempt at data-centric optimization for inference efficiency, and showcases the potential of eliciting high-quality answers by explicitly planning the answer structure in language.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 103 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

37 of 103 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  2. 2022Large Language Models Can Self-Improve20 Oct

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2020Reformer: The Efficient Transformercited by 1,658
23 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  13. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  15. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  16. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  17. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  18. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  19. 2021Accelerating Sparse Deep Neural Networkscited by 213
  20. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  21. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  22. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  23. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 46; 2026: 38.

103 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  3. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  4. 2025Efficient Reasoning Models: A Surveycited by 93
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  7. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  8. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  9. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  10. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  11. 2025VLM-R3: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thoughtcited by 47
  12. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
38 more
  1. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  2. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  3. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  4. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  5. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  6. 2023Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Trainingcited by 21
  7. 2024K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoningcited by 21
  8. 2024ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarizationcited by 19
  9. 2024Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matchingcited by 18
  10. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  11. 2025The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Modelscited by 17
  12. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  13. 2025MASTER: A Multi-Agent System with LLM Specialized MCTScited by 15
  14. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 15
  15. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  16. 2024An LLM-Tool Compiler for Fused Parallel Function Callingcited by 14
  17. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  18. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  19. 2025Self-Steering Language Modelscited by 13
  20. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decodingcited by 11
  21. 2025Hawkeye:Efficient Reasoning with Model Collaborationcited by 11
  22. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  23. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMscited by 11
  24. 2025Towards Transparent and Incentive-Compatible Collaboration in Decentralized LLM Multi-Agent Systems: A Blockchain-Driven Approachcited by 11
  25. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  26. 2024Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Informationcited by 10
  27. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  28. 2025SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Modelscited by 10
  29. 2025FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edgecited by 9
  30. 2025OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Drivingcited by 9
  31. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  32. 2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
  33. 2025Planned Diffusioncited by 8
  34. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inferencecited by 8
  35. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  36. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  37. 2026DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviorscited by 7
  38. 2026Decentralized Multi-Agent Systems with Shared Contextcited by 7

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Plato: Plan to Efficiently Decode for Large Language Model Inference
  2. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning
  3. 2024Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs
  4. 2025Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately
  5. 2026Self-Speculation for Faster Reasoning Models
  6. 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Models
  7. 2025Early Stopping Chain-of-thoughts in Large Language Models
  8. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling