mlsanityData to 14 Oct 2026

LPZero: Language Model Zero-cost Proxy Search from Zero

arXiv 2410.04808 ↗First posted 7 Oct 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

In spite of the outstanding performance, Neural Architecture Search (NAS) is criticized for massive computation. Recently, Zero-shot NAS has emerged as a promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce computational demands. Despite this, existing ZC proxies heavily rely on expert knowledge and incur significant trial-and-error costs. Particularly in NLP tasks, most existing ZC proxies fail to surpass the performance of the naive baseline. To address these challenges, we introduce a novel framework, LPZero, which is the first to automatically design ZC proxies for various tasks, achieving higher ranking consistency than human-designed proxies. Specifically, we model the ZC proxy as a symbolic equation and incorporate a unified proxy search space that encompasses existing ZC proxies, which are composed of a predefined set of mathematical symbols. To heuristically search for the best ZC proxy, LPZero incorporates genetic programming to find the optimal symbolic composition. We propose a textitRule-based Pruning Strategy (RPS), which preemptively eliminates unpromising proxies, thereby mitigating the risk of proxy degradation. Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's superior ranking ability and performance on downstream tasks compared to current approaches.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

45 of 59 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
33 more
  1. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  2. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  5. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  6. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  7. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  8. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  11. 2020Neural Architecture Search without Trainingcited by 244
  12. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  13. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  14. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  15. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  16. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  17. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  18. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  19. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 73
  20. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  21. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  22. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  23. 2019AM-LFS: AutoML for Loss Function Searchcited by 29
  24. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  25. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26
  26. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  27. 2021AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Taskscited by 19
  28. 2020Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentationcited by 16
  29. 2022EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoringcited by 15
  30. 2022LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Modelscited by 14
  31. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13
  32. 2023Training-free Neural Architecture Search for RNNs and Transformerscited by 9
  33. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  2. 2025EvoLattice: Persistent Internal-Population Evolution through Multi-Alternative Quality-Diversity Graph Representations for LLM-Guided Program Discoverycited by 3
  3. 2025ZeroLM: Data-Free Transformer Architecture Search for Language Modelscited by 2
  4. 2025Cognitive-YOLO: LLM-Driven Architecture Synthesis from First Principles of Data for Object Detectioncited by 1
  5. 2026TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlationcited by 1
  6. 2025HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inferencecited by 0
  7. 2026ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Searchcited by 0