Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector
Not on the map. Why?
Abstract (arXiv)
Large language models have demonstrated impressive capabilities across various domains. However, their general-purpose nature often limits their effectiveness in specialized fields such as energy, where deep technical expertise and precise domain knowledge are essential. In this paper, we introduce EnergyGPT, a domain-specialized language model tailored for the energy sector, developed by fine-tuning the LLaMA 3.1-8B model on a high-quality, curated corpus of energy-related texts. We consider two adaptation strategies: a full-parameter Supervised Fine-Tuning variant and a parameter-efficient LoRA-based variant that updates only a small fraction of the model parameters. We present a complete development pipeline, including data collection and curation, model fine-tuning, benchmark design and LLM-judge choice, evaluation, and deployment. Through this work, we demonstrate that our training strategy enables improvements in domain relevance and performance without the need for large-scale infrastructure. By evaluating the performance of both EnergyGPT variants using domain-specific question-answering benchmarks, our results show that the adapted models consistently outperform the base model in most energy-related language understanding and generation tasks, with the LoRA variant achieving competitive gains at significantly reduced training cost.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 53 of its 87 references that are in the map. First posted on arXiv 8 Sep 2025.
Builds on
53 of 87 references.
References · 53
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
38 more
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2023BloombergGPT: A Large Language Model for Financecited by 939
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2022Galactica: A Large Language Model for Sciencecited by 744
- 2021Deduplicating Training Data Makes Language Models Bettercited by 714
- 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
- 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
- 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
- 2023Scaling Data-Constrained Language Modelscited by 403
- 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
- 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
- 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
- 2023Continual Pre-training of Language Modelscited by 198
- 2021Towards Continual Knowledge Learning of Language Modelscited by 174
- 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
- 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 139
- 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
- 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
- 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
- 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
- 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
- 2024The rising costs of training frontier AI modelscited by 95
- 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
- 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 90
- 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
- 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
- 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
- 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
- 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 58
- 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 40
- 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
- 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 37
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.