mlsanityData to 14 Oct 2026

Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Software

arXiv 2502.03916 ↗First posted 6 Feb 2025cs.CL cs.AIcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Large Language Models (LLMs) are tools that have become indispensable in development and programming. However, they suffer from hallucinations, especially when dealing with unknown knowledge. This is particularly the case when LLMs are to be used to support closed-source software applications. Retrieval-Augmented Generation (RAG) offers an approach to use additional knowledge alongside the pre-trained knowledge of the LLM to respond to user prompts. Possible tasks range from a smart-autocomplete, text extraction for question answering, model summarization, component explaining, compositional reasoning, to creation of simulation components and complete input models. This work tests existing RAG systems for closed-source simulation frameworks, in our case the mesh-free simulation software Pasimodo. Since data protection and intellectual property rights are particularly important for problems solved with closed-source software, the tests focus on execution using local LLMs. In order to enable smaller institutions to use the systems, smaller language models will be tested first. The systems show impressive results, but often fail due to insufficient information. Different approaches for improving response quality are tested. In particular, tailoring the information provided to the LLMs dependent to the prompts proves to be a significant improvement. This demonstrates the great potential and the further work needed to improve information retrieval for closed-source simulation models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 65 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

11 of 65 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  8. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  9. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 93
  10. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  11. 2024Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 3
  2. 2026Large Language Models and their Awareness of Mechanics and Spatial Geometrycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  2. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  3. 2024Deploying Large Language Models With Retrieval Augmented Generation
  4. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  5. 2025Post-training an LLM for RAG? Train on Self-Generated Demonstrations
  6. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
  7. 2025SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA
  8. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG