# FeynTune: Large Language Models for High-Energy Theory

arXiv: https://arxiv.org/abs/2508.03716 · HTML: https://mlsanity.com/paper/2508.03716/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [AstroLLaMA: Towards Specialized Foundation Models in Astronomy](https://mlsanity.com/paper/2309.06126/) (2023)
- [Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physics](https://mlsanity.com/paper/2502.15815/) (2025)
- [Building astroBERT, a language model for Astronomy & Astrophysics](https://mlsanity.com/paper/2112.00590/) (2021)
- [SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity](https://mlsanity.com/paper/2401.17072/) (2024)
- [AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets](https://mlsanity.com/paper/2401.01916/) (2024)
- [PhysBERT: A Text Embedding Model for Physics Scientific Literature](https://mlsanity.com/paper/2408.09574/) (2024)
- [AstroMLab 1: Who Wins Astronomy Jeopardy!?](https://mlsanity.com/paper/2407.11194/) (2024)

## Built on by (most cited first, 3 of 3)

- [gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy](https://mlsanity.com/paper/2605.11269/) (2026)
- [Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMs](https://mlsanity.com/paper/2604.14188/) (2026)
- [SuperAI: Merging Capabilities Across Heterogeneous Foundation Models](https://mlsanity.com/paper/2610.15322/) (2026)

## Abstract (arXiv)

We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.