# From Transformers to Large Language Models: A systematic review of AI applications in the energy sector towards Agentic Digital Twins

arXiv: https://arxiv.org/abs/2506.06359 · HTML: https://mlsanity.com/paper/2506.06359/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 126 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [TimeGPT-1](https://mlsanity.com/paper/2310.03589/) (2023)
- [A Survey on Large Language Models with some Insights on their Capabilities and Limitations](https://mlsanity.com/paper/2501.04040/) (2025)
- [ChatGPT and Other Large Language Models for Cybersecurity of Smart Grid Applications](https://mlsanity.com/paper/2311.05462/) (2023)
- [The Role of LLMs in Sustainable Smart Cities: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2402.14596/) (2024)
- [Applying Large Language Models to Power Systems: Potential Security Threats](https://mlsanity.com/paper/2311.13361/) (2023)
- [Massively Digitized Power Grid: Opportunities and Challenges of Use-inspired AI](https://mlsanity.com/paper/2205.05180/) (2022)
- [Leveraging Large Language Models for Enhanced Digital Twin Modeling: Trends, Methods, and Challenges](https://mlsanity.com/paper/2503.02167/) (2025)
- [Advancing Building Energy Modeling with Large Language Models: Exploration and Case Studies](https://mlsanity.com/paper/2402.09579/) (2024)
- [Dynamic Rolling Horizon Optimization for Network-Constrained V2X Value Stacking of Electric Vehicles Under Uncertainties](https://mlsanity.com/paper/2502.09290/) (2025)

## Built on by (most cited first, 2 of 2)

- [When Agents Meet Electric Bus Fleet Operations: Pricing Behavior, Trade-offs, and Policy Implications in an Aggregator Framework](https://mlsanity.com/paper/2606.26400/) (2026)
- [A Multimodal Dataset for Large Language Model Applications in the Energy Domain](https://mlsanity.com/paper/2607.11459/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) has long promised to improve energy management in smart grids by enhancing situational awareness and supporting more effective decision-making. While traditional machine learning has demonstrated notable results in forecasting and optimization, it often struggles with generalization, situational awareness, and heterogeneous data integration. Recent advances in foundation models such as Transformer architecture and Large Language Models (LLMs) have demonstrated improved capabilities in modelling complex temporal and contextual relationships, as well as in multi-modal data fusion which is essential for most AI applications in the energy sector. In this review we synthesize the rapid expanding field of AI applications in the energy domain focusing on Transformers and LLMs. We examine the architectural foundations, domain-specific adaptations and practical implementations of transformer models across various forecasting and grid management tasks. We then explore the emerging role of LLMs in the field: adaptation and fine tuning for the energy sector, the type of tasks they are suited for, and the new challenges they introduce. Along the way, we highlight practical implementations, innovations, and areas where the research frontier is rapidly expanding. These recent developments reviewed underscore a broader trend: Generative AI (GenAI) is beginning to augment decision-making not only in high-level planning but also in day-to-day operations, from forecasting and grid balancing to workforce training and asset onboarding. Building on these developments, we introduce the concept of the Agentic Digital Twin, a next-generation model that integrates LLMs to bring autonomy, proactivity, and social interaction into digital twin-based energy management systems.