mlsanityData to 7 Oct 2026

Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey

arXiv 2505.03418 ↗First posted 6 May 2025cs.LGcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Problem-solving has been a fundamental driver of human progress in numerous domains. With advancements in artificial intelligence, Large Language Models (LLMs) have emerged as powerful tools capable of tackling complex problems across diverse domains. Unlike traditional computational systems, LLMs combine raw computational power with an approximation of human reasoning, allowing them to generate solutions, make inferences, and even leverage external computational tools. However, applying LLMs to real-world problem-solving presents significant challenges, including multi-step reasoning, domain knowledge integration, and result verification. This survey explores the capabilities and limitations of LLMs in complex problem-solving, examining techniques including Chain-of-Thought (CoT) reasoning, knowledge augmentation, and various LLM-based and tool-based verification techniques. Additionally, we highlight domain-specific challenges in various domains, such as software engineering, mathematical reasoning and proving, data analysis and modeling, and scientific research. The paper further discusses the fundamental limitations of the current LLM solutions and the future directions of LLM-based complex problems solving from the perspective of multi-step reasoning, domain knowledge integration and result verification.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 103 of its 157 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

103 of 157 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling29 Aug
  2. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuning28 Feb

Other references · 101

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2024A Survey on LLM-as-a-Judgecited by 1,384
38 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  3. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2024A Survey on Large Language Models for Code Generationcited by 646
  12. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  13. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  14. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  15. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  16. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  17. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  18. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  19. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  20. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  21. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  22. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  23. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  24. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  25. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  26. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  27. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  28. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  29. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  30. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  31. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 174
  32. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  33. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  34. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  35. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  36. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  37. 2025LIMR: Less is More for RL Scalingcited by 136
  38. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 69
  2. 2025AutoMind: Adaptive Knowledgeable Agent for Automated Data Sciencecited by 20
  3. 2025Autoformalization in the Era of Large Language Models: A Surveycited by 19
  4. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  5. 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflectioncited by 3
  6. 2025Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reportscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  2. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey
  3. 2025Large Language Models and Mathematical Reasoning Failures
  4. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  5. 2025Computational Thinking Reasoning in Large Language Models
  6. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  7. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Models
  8. 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Review