mlsanityData to 14 Oct 2026

A Method for Parsing and Vectorization of Semi-structured Data used in Retrieval Augmented Generation

arXiv 2405.03989 ↗First posted 7 May 2024cs.DBcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

This paper presents a novel method for parsing and vectorizing semi-structured data to enhance the functionality of Retrieval-Augmented Generation (RAG) within Large Language Models (LLMs). We developed a comprehensive pipeline for converting various data formats into .docx, enabling efficient parsing and structured data extraction. The core of our methodology involves the construction of a vector database using Pinecone, which integrates seamlessly with LLMs to provide accurate, context-specific responses, particularly in environmental management and wastewater treatment operations. Through rigorous testing with both English and Chinese texts in diverse document formats, our results demonstrate a marked improvement in the precision and reliability of LLMs outputs. The RAG-enhanced models displayed enhanced ability to generate contextually rich and technically accurate responses, underscoring the potential of vector knowledge bases in significantly boosting the performance of LLMs in specialized domains. This research not only illustrates the effectiveness of our method but also highlights its potential to revolutionize data processing and analysis in environmental sciences, setting a precedent for future advancements in AI-driven applications. Our code is available at https://github.com/linancn/TianGong-AI-Unstructure.git.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 3 of its 3 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

3 of 3 references.

References · 3

  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Optimizing Retrieval-Augmented Generation for Electrical Engineering: A Case Study on ABB Circuit Breakerscited by 0
  2. 2025MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report
  2. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation
  5. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  6. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey
  7. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  8. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation