mlsanityData to 7 Oct 2026

Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning

arXiv 2312.08901 ↗First posted 14 Dec 2023cs.CL cs.AIcited by 26

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive capabilities, yet they still struggle with math reasoning. In this work, we propose CoT-Influx, a novel approach that pushes the boundary of few-shot Chain-of-Thoughts (CoT) learning to improve LLM mathematical reasoning. Motivated by the observation that adding more concise CoT examples in the prompt can improve LLM reasoning performance, CoT-Influx employs a coarse-to-fine pruner to maximize the input of effective and concise CoT examples. The pruner first selects as many crucial CoT examples as possible and then prunes unimportant tokens to fit the context window. A math reasoning dataset with diverse difficulty levels and reasoning steps is used to train the pruner, along with a math-specialized reinforcement learning approach. As a result, by enabling more CoT examples with double the context window size in tokens, CoT-Influx significantly outperforms various prompting baselines across various LLMs (LLaMA2-7B, 13B, 70B) and 5 math datasets, achieving up to 4.55% absolute improvements. Remarkably, without any fine-tuning, LLaMA2-70B with CoT-Influx surpasses GPT-3.5 and a wide range of larger LLMs (PaLM, Minerva 540B, etc.) on the GSM8K. CoT-Influx serves as a plug-and-play module for LLMs and is compatible with most existing reasoning prompting techniques, such as self-consistency and self-verification.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 60 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

32 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation7 Nov
  2. 2022Large Language Models are Better Reasoners with Self-Verification19 Dec

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
18 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  3. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  4. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  5. 2023Adapting Language Models to Compress Contextscited by 350
  6. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  7. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  8. 2023Augmenting Language Models with Long-Term Memorycited by 186
  9. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  10. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  11. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  12. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  13. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  14. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  15. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  16. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  17. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  18. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  5. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  7. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2024TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learningcited by 22
  10. 2024Neuro-Symbolic Data Generation for Math Reasoningcited by 21
  11. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  12. 2025Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidencecited by 8
14 more
  1. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  2. 2025Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Modelscited by 7
  3. 2025CoTSRF: Utilize Chain of Thought as Stealthy and Robust Fingerprint of Large Language Modelscited by 7
  4. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  5. 2024EF-LLM: Energy Forecasting LLM with AI-assisted Automation, Enhanced Sparse Prediction, Hallucination Detectioncited by 6
  6. 2025EFPC: Towards Efficient and Flexible Prompt Compressioncited by 5
  7. 2025GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analyticscited by 4
  8. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  9. 2025EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensationcited by 2
  10. 2026LLMs with in-context learning for Algorithmic Theoretical Physicscited by 2
  11. 2024Evolutionary Pre-Prompt Optimization for Mathematical Reasoningcited by 1
  12. 2025GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoningcited by 0
  13. 2025An Empirical Framework for Evaluating Semantic Preservation Using Hugging Facecited by 0
  14. 2026Generating Public Health Responses using Survey-Augmented Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
  2. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
  3. 2025A Survey on Large Language Models for Mathematical Reasoning
  4. 2025CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
  5. 2025Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoning
  6. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
  7. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning
  8. 2022Large Language Models are Zero-Shot Reasoners