mlsanityData to 14 Oct 2026

Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving

arXiv 2510.00919 ↗First posted 1 Oct 2025cs.CL cs.AIcited by 1

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

Retrieval-augmented generation (RAG) with foundation models has achieved strong performance across diverse tasks, but their capacity for expert-level reasoning-such as solving Olympiad-level physics problems-remains largely unexplored. Inspired by the way students prepare for competitions by reviewing past problems, we investigate the potential of RAG to enhance physics reasoning in foundation models. We introduce PhoPile, a high-quality multimodal dataset specifically designed for Olympiad-level physics, enabling systematic study of retrieval-based reasoning. PhoPile includes diagrams, graphs, and equations, capturing the inherently multimodal nature of physics problem solving. Using PhoPile, we benchmark RAG-augmented foundation models, covering both large language models (LLMs) and large multimodal models (LMMs) with multiple retrievers. Our results demonstrate that integrating retrieval with physics corpora can improve model performance, while also highlighting challenges that motivate further research in retrieval-augmented physics reasoning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 46 of its 77 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

46 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Models27 Jun

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
33 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  14. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  15. 2023Llemma: An Open Language Model For Mathematicscited by 388
  16. 2022Teaching Small Language Models to Reasoncited by 284
  17. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  18. 2022In-context Examples Selection for Machine Translationcited by 172
  19. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  20. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  21. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 118
  22. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 111
  23. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  24. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  25. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  26. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  27. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 52
  28. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 49
  29. 2024Large Language Models Are Neurosymbolic Reasonerscited by 31
  30. 2023Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Gamecited by 28
  31. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?cited by 12
  32. 2023MLFMF: Data Sets for Machine Learning for Mathematical Formalizationcited by 10
  33. 2025PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environmentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
  2. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level
  3. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
  4. 2026MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
  5. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
  6. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
  7. 2025Symbolic or Numerical? Understanding Physics Problem Solving in Reasoning LLMs
  8. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning