# LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models

arXiv: https://arxiv.org/abs/2405.18377 · HTML: https://mlsanity.com/paper/2405.18377/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities](https://mlsanity.com/paper/2205.10358/) (2022)
- [InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning](https://mlsanity.com/paper/2308.15609/) (2023)

## Built on by (most cited first, 9 of 9)

- [Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques](https://mlsanity.com/paper/2505.02309/) (2025)
- [ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs](https://mlsanity.com/paper/2503.10714/) (2025)
- [A Simple Linear Patch Revives Layer-Pruned Large Language Models](https://mlsanity.com/paper/2505.24680/) (2025)
- [LLMs as Debate Partners: Utilizing Genetic Algorithms and Adversarial Search for Adaptive Arguments](https://mlsanity.com/paper/2412.06229/) (2024)
- [Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs](https://mlsanity.com/paper/2507.02909/) (2025)
- [Bilevel Optimization for Neural Architecture Search](https://mlsanity.com/paper/2606.29582/) (2026)
- [TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation](https://mlsanity.com/paper/2602.02891/) (2026)
- [Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning](https://mlsanity.com/paper/2501.05248/) (2025)
- [LP-NAS: Linear Programming-based Neural Architecture Search](https://mlsanity.com/paper/2608.14472/) (2026)

## Abstract (arXiv)

The abilities of modern large language models (LLMs) in solving natural language processing, complex reasoning, sentiment analysis and other tasks have been extraordinary which has prompted their extensive adoption. Unfortunately, these abilities come with very high memory and computational costs which precludes the use of LLMs on most hardware platforms. To mitigate this, we propose an effective method of finding Pareto-optimal network architectures based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B only once and then apply genetic algorithm-based search to find smaller, less computationally complex network architectures. We show that, for certain standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily large and complex. More specifically, we demonstrate a 1.5x reduction in model size and 1.3x speedup in throughput for certain tasks with negligible drop in accuracy. In addition to finding smaller, higher-performing network architectures, our method does so more effectively and efficiently than certain pruning or sparsification techniques. Finally, we demonstrate how quantization is complementary to our method and that the size and complexity of the networks we find can be further decreased using quantization. We believe that our work provides a way to automatically create LLMs which can be used on less expensive and more readily available hardware platforms.