# Comparative Study of Large Language Model Architectures on Frontier

arXiv: https://arxiv.org/abs/2402.00691 · HTML: https://mlsanity.com/paper/2402.00691/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2024.

## Built on by (most cited first, 4 of 4)

- [34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery](https://mlsanity.com/paper/2505.03049/) (2025)
- [Demystifying the Communication Characteristics for Distributed Transformer Models](https://mlsanity.com/paper/2408.10197/) (2024)
- [Sequence Length Scaling in Vision Transformers for Scientific Images on Frontier](https://mlsanity.com/paper/2405.15780/) (2024)
- [Uncovering Competency Gaps in Large Language Models and Their Benchmarks](https://mlsanity.com/paper/2512.20638/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have garnered significant attention in both the AI community and beyond. Among these, the Generative Pre-trained Transformer (GPT) has emerged as the dominant architecture, spawning numerous variants. However, these variants have undergone pre-training under diverse conditions, including variations in input data, data preprocessing, and training methodologies, resulting in a lack of controlled comparative studies. Here we meticulously examine two prominent open-sourced GPT architectures, GPT-NeoX and LLaMA, leveraging the computational power of Frontier, the world's first Exascale supercomputer. Employing the same materials science text corpus and a comprehensive end-to-end pipeline, we conduct a comparative analysis of their training and downstream performance. Our efforts culminate in achieving state-of-the-art performance on a challenging materials science benchmark. Furthermore, we investigate the computation and energy efficiency, and propose a computationally efficient method for architecture design. To our knowledge, these pre-trained models represent the largest available for materials science. Our findings provide practical guidance for building LLMs on HPC platforms.