mlsanityData to 14 Oct 2026

FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning

arXiv 2601.19001 ↗First posted 26 Jan 2026cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose FROST, an attention-aware method for efficient reasoning. Unlike traditional approaches, FROST leverages attention weights to prune uncritical reasoning paths, yielding shorter and more reliable reasoning trajectories. Methodologically, we introduce the concept of reasoning outliers and design an attention-based mechanism to remove them. Theoretically, FROST preserves and enhances the model's reasoning capacity while eliminating outliers at the sentence level. Empirically, we validate FROST on four benchmarks using two strong reasoning models (Phi-4-Reasoning and GPT-OSS-20B), outperforming state-of-the-art methods such as TALE and ThinkLess. Notably, FROST achieves an average 69.68% reduction in token usage and a 26.70% improvement in accuracy over the base model. Furthermore, in evaluations of attention outlier metrics, FROST reduces the maximum infinity norm by 15.97% and the average kurtosis by 91.09% compared to the base model. Code is available at https://github.com/robinzixuan/FROST

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 59 of its 75 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

59 of 75 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  9. 2024Phi-4 Technical Reportcited by 711
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  12. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  13. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  14. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 297
  15. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 249
  16. 2024Token-Budget-Aware LLM Reasoningcited by 247
  17. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  18. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 169
  19. 2019Adaptively Sparse Transformerscited by 155
  20. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
  21. 2025Phi-4-reasoning Technical Reportcited by 151
  22. 2025Thinkless: LLM Learns When to Thinkcited by 113
  23. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  24. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 96
  25. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  26. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 72
  27. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  28. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  29. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 54
  30. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 48
  31. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 48
  32. 2023On Sparse Modern Hopfield Modelcited by 42
  33. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  34. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 30
  35. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
  36. 2024Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Modelscited by 27
  37. 2024SemCoder: Training Code Language Models with Comprehensive Semantics Reasoningcited by 27
  38. 2025Let LRMs Break Free from Overthinking via Self-Braking Tuningcited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  2. 2026Attention Sinks and Outliers in Attention Residualscited by 2
  3. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimizationcited by 2
  4. 2026ThoughtFold: Folding Reasoning Chains via Introspective Preference Learningcited by 1
  5. 2026Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
  2. 2025Efficient Inference for Large Reasoning Models: A Survey
  3. 2025FlashThink: An Early Exit Method For Efficient Reasoning
  4. 2026Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Models
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
  6. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval
  7. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models
  8. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning