# Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness

arXiv: https://arxiv.org/abs/2408.04585 · HTML: https://mlsanity.com/paper/2408.04585/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Efficient Adversarial Training in LLMs with Continuous Attacks](https://mlsanity.com/paper/2405.15589/) (2024)
- ["Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models](https://mlsanity.com/paper/2406.19417/) (2024)
- [Adversarial Attacks and Defenses in Large Language Models: Old and New Threats](https://mlsanity.com/paper/2310.19737/) (2023)
- [ComposerX: Multi-Agent Symbolic Music Composition with LLMs](https://mlsanity.com/paper/2404.18081/) (2024)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [AI-powered Code Review with LLMs: Early Results](https://mlsanity.com/paper/2404.18496/) (2024)
- [The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative](https://mlsanity.com/paper/2402.14859/) (2024)
- [Systematic Rectification of Language Models via Dead-end Analysis](https://mlsanity.com/paper/2302.14003/) (2023)

## Built on by (most cited first, 22 of 22)

- [Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation](https://mlsanity.com/paper/2409.11703/) (2024)
- [NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training](https://mlsanity.com/paper/2409.09582/) (2024)
- [AR Overlay: Training Image Pose Estimation on Curved Surface in a Synthetic Way](https://mlsanity.com/paper/2409.14577/) (2024)
- [Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review](https://mlsanity.com/paper/2410.03663/) (2024)
- [CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing](https://mlsanity.com/paper/2410.16670/) (2024)
- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [SteerDiff: Steering towards Safe Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.02710/) (2024)
- [LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuning](https://mlsanity.com/paper/2411.09947/) (2024)
- [Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs](https://mlsanity.com/paper/2412.14304/) (2024)
- [Enhanced Recommendation Combining Collaborative Filtering and Large Language Models](https://mlsanity.com/paper/2412.18713/) (2024)
- [Style Transfer: From Stitching to Neural Networks](https://mlsanity.com/paper/2409.00606/) (2024)
- [Predicting 30-Day Hospital Readmission in Medicare Patients: Insights from an LSTM Deep Learning Model](https://mlsanity.com/paper/2410.17545/) (2024)
- [Comprehensive Overview of Artificial Intelligence Applications in Modern Industries](https://mlsanity.com/paper/2409.13059/) (2024)
- [MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation](https://mlsanity.com/paper/2503.17900/) (2025)
- [EvalCards: A Framework for Standardized Evaluation Reporting](https://mlsanity.com/paper/2511.21695/) (2025)
- [Using Large Language Models to Generate Clinical Trial Tables and Figures](https://mlsanity.com/paper/2409.12046/) (2024)
- [Enhanced Credit Score Prediction Using Ensemble Deep Learning Model](https://mlsanity.com/paper/2410.00256/) (2024)
- [Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models](https://mlsanity.com/paper/2410.19241/) (2024)
- [Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System](https://mlsanity.com/paper/2505.01315/) (2025)
- [Context-Aware Token Selection and Packing for Enhanced Vision Transformer](https://mlsanity.com/paper/2410.23608/) (2024)
- [A Survey: Towards Privacy and Security in Mobile Large Language Models](https://mlsanity.com/paper/2509.02411/) (2025)
- [Beyond the Benchmark: Innovative Defenses Against Prompt Injection Attacks](https://mlsanity.com/paper/2512.16307/) (2025)

## Abstract (arXiv)

With the increasing demand for practical applications of Large Language Models (LLMs), many attention-efficient models have been developed to balance performance and computational cost. However, the adversarial robustness of these models remains under-explored. In this work, we design a framework to investigate the trade-off between efficiency, performance, and adversarial robustness of LLMs and conduct extensive experiments on three prominent models with varying levels of complexity and efficiency -- Transformer++, Gated Linear Attention (GLA) Transformer, and MatMul-Free LM -- utilizing the GLUE and AdvGLUE datasets. The AdvGLUE dataset extends the GLUE dataset with adversarial samples designed to challenge model robustness. Our results show that while the GLA Transformer and MatMul-Free LM achieve slightly lower accuracy on GLUE tasks, they demonstrate higher efficiency and either superior or comparative robustness on AdvGLUE tasks compared to Transformer++ across different attack levels. These findings highlight the potential of simplified architectures to achieve a compelling balance between efficiency, performance, and adversarial robustness, offering valuable insights for applications where resource constraints and resilience to adversarial attacks are critical.