mlsanityData to 7 Oct 2026

From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models

arXiv 2406.16838 ↗First posted 24 Jun 2024cs.CL cs.LGcited by 136

Not on the map. Why?

Cited by 78 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One of the most striking findings in modern research on large language models (LLMs) is that scaling up compute during training leads to better results. However, less attention has been given to the benefits of scaling compute during inference. This survey focuses on these inference-time approaches. We explore three areas under a unified mathematical formalism: token-level generation algorithms, meta-generation algorithms, and efficient generation. Token-level generation algorithms, often called decoding algorithms, operate by sampling a single token at a time or constructing a token-level search space and then selecting an output. These methods typically assume access to a language model's logits, next-token distributions, or probability scores. Meta-generation algorithms work on partial or full sequences, incorporating domain knowledge, enabling backtracking, and integrating external information. Efficient generation methods aim to reduce token costs and improve the speed of generation. Our survey unifies perspectives from three research communities: traditional natural language processing, modern LLMs, and machine learning systems.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 140 of its 190 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

140 of 190 references.

References · 140

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  14. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  15. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  16. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  17. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  18. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  19. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  20. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  21. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  22. 2018Hierarchical Neural Story Generationcited by 1,182
  23. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  24. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  25. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  26. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  27. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  28. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  29. 2023Teaching Large Language Models to Self-Debugcited by 963
  30. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  31. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  32. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  33. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  34. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  35. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  36. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  37. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  38. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598

50 most cited of 140 All 140 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 83; 2026: 34.

136 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  3. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  4. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2024AI Agents That Mattercited by 199
  7. 2025Deep Think with Confidencecited by 199
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  10. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  11. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  12. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
38 more
  1. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  3. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  4. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  5. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  6. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  7. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  8. 2025The Majority is not always right: RL training for solution aggregationcited by 46
  9. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  10. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  11. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
  12. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  13. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  14. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  15. 2024InfAlign: Inference-aware language model alignmentcited by 34
  16. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 33
  17. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  18. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  19. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  20. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  21. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  22. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  23. 2025Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMscited by 22
  24. 2025Train Long, Think Short: Curriculum Learning for Efficient Reasoningcited by 20
  25. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  26. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  27. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  28. 2024Scaling LLM Inference with Optimized Sample Compute Allocationcited by 16
  29. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  30. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  31. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  32. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  33. 2025Enhancing Latent Computation in Transformers with Latent Tokenscited by 13
  34. 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracycited by 13
  35. 2025Scaling Speculative Decoding with Lookahead Reasoningcited by 13
  36. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  37. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  38. 2024Plan*RAG: Efficient Test-Time Planning for Retrieval Augmented Generationcited by 11

50 most cited of 136 All 136 (JSON)