mlsanityData to 14 Oct 2026

Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs

arXiv 2605.04903 ↗First posted 6 May 2026cs.LG cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) show strong potential for neural architecture generation, yet existing approaches produce complete model implementations from scratch -- computationally expensive and yielding verbose code. We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs (deltas) to refine baseline architectures rather than synthesizing entire models. Our pipeline iteratively fine-tunes the LLM via LoRA on curated architectures from the LEMUR dataset, with MinHash-Jaccard novelty filtering for structural diversity. We evaluate three 7B-class LLMs -- DeepSeek-Coder-7B, Qwen2.5-Coder-7B, and Mistral-7B -- across six datasets (CIFAR-10, CIFAR-100, MNIST, SVHN, ImageNette, CelebA) using a 22-cycle protocol (1,100 candidates per LLM). All three substantially surpass the full-generation baseline (50.6% valid rate, 42.3% mean first-epoch accuracy): DeepSeek-Coder reaches 75.3% valid rate and 65.8% mean accuracy; Qwen2.5-Coder 72.1%/64.6%; Mistral 66.6%/66.1%. On CIFAR-10, best first-epoch accuracies reach 85.5% (Mistral), 85.2% (DeepSeek), 80.6% (Qwen) -- well above 63.98% full generation and 71.5% for the concurrent approach of Gu et al. Output lengths are 30-50 lines versus 200+ for full generation (75-85% reduction). A 50-epoch study confirms the 1-epoch proxy preserves rankings (Mistral: Spearman ρ = 0.926). Delta-based generation is a token-efficient, multi-domain, LLM-agnostic alternative to full-model synthesis for LLM-driven NAS.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
29 more
  1. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  4. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  5. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  6. 2020Neural Architecture Search without Trainingcited by 244
  7. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  8. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  9. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  10. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 87
  11. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  12. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  13. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  14. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  15. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  16. 2025LEMUR Neural Network Dataset: Towards Seamless AutoMLcited by 20
  17. 2025NNGPT: Rethinking AutoML with Large Language Modelscited by 13
  18. 2024SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAScited by 11
  19. 2025VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?cited by 11
  20. 2025Preparation of Fractal-Inspired Computational Architectures for Advanced Large Language Model Analysiscited by 11
  21. 2025A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networkscited by 11
  22. 2025AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance?cited by 10
  23. 2025Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Designcited by 10
  24. 2026From Memorization to Creativity: LLM as a Designer of Novel Neural Architecturescited by 10
  25. 2025LLM as a Neural Architect: Controlled Generation of Image Captioning Models Under Strict API Contractscited by 9
  26. 2025SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Modelscited by 7
  27. 2026From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMscited by 7
  28. 2026Resource-Efficient Iterative LLM-Based NAS with Feedback Memorycited by 7
  29. 2026Closed-Loop LLM Discovery of Non-Standard Channel Priors in Vision Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliabilitycited by 1
  2. 2026Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Searchcited by 1
  3. 2026Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Studycited by 0