mlsanityData to 14 Oct 2026

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain

arXiv 2609.15242 ↗First posted 14 Sep 2026cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Environmental, Social, and Governance (ESG) reporting is critical for corporate accountability, with Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) offering strong potential to automate KPI extraction. However, open-source LLM performance in domain-specific ESG tasks remains insufficiently understood. This paper evaluates open-source LLMs in ESG contexts using a structured framework and evaluation resource based on 498 real-world ESG reports from EU-listed companies (2010-2024). We evaluate seven open-source models (2B to 30B parameters) -- glm-4.7-flash, nemotron-3-nano:4b, qwen3:4b-instruct, gemma3:4b, gemma4:e4b, gemma4:e2b, and ministral-3:8b -- using 100 persona-based synthetic QA pairs covering ESG information needs. System performance is assessed via RAGAS metrics, including contextual recall, precision, relevance, faithfulness, answer relevancy, and factual correctness. Results show notable performance variations across architectures. Retrieval performance is strong across models (context recall around 0.58-0.61, context precision around 0.78-0.81, context relevance 0.965-0.985). Generation diverges most on faithfulness (0.607-0.822) and least on answer relevancy (0.760-0.881): glm-4.7-flash leads in faithfulness (0.822), qwen3 in factual correctness (0.449), and ministral-3 in answer relevancy (0.881). Low overall factual correctness (0.387-0.449) highlights the need for domain-specific fine-tuning. This work provides data-driven guidance for deploying open-source models in ESG reporting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 43 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

22 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems29 Nov
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  12. 2026Ministral 3cited by 114
7 more
  1. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  2. 2023ESGReveal: An LLM-based approach for extracting structured data from ESG reportscited by 15
  3. 2025ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledgecited by 8
  4. 2024SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generationcited by 7
  5. 2025Optimizing Large Language Models for ESG Activity Detection in Financial Textscited by 5
  6. 2025Enhancing Retrieval for ESGLLM via ESG-CID -- A Disclosure Content Index Finetuning Dataset for Mapping GRI and ESRScited by 2
  7. 2026ESGLens: An LLM-Based RAG Framework for Interactive ESG Report Analysis and Score Predictioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability
  2. 2026Benchmarking Embedding Models for ESG Data ↗
  3. 2024Deploying Large Language Models With Retrieval Augmented Generation
  4. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
  5. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  6. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
  7. 2025Know Or Not: a library for evaluating out-of-knowledge base robustness
  8. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation