mlsanityData to 7 Oct 2026

Investigating the Effectiveness of HyperTuning via Gisting

arXiv 2402.16817 ↗First posted 26 Feb 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-based hypernetworks. We introduce HyperLlama, a set of Gisting-based hypernetworks built on Llama-2 models that generates task-specific soft prefixes based on few-shot inputs. In experiments across P3, Super-NaturalInstructions and Symbol Tuning datasets, we show that HyperLlama models can effectively compress information from few-shot examples into soft prefixes. However, they still underperform multi-task fine-tuned language models with full attention over few-shot in-context examples. We also show that HyperLlama-generated soft prefixes can serve as better initializations for further prefix tuning. Overall, Gisting-based hypernetworks are economical and easy to implement, but have mixed empirical performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

24 of 26 references.

References · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2016HyperNetworkscited by 1,003
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022Transformers learn in-context by gradient descentcited by 714
12 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  2. 2021MetaICL: Learning to Learn In Contextcited by 450
  3. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  4. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  5. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  6. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  7. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  8. 2023Symbol tuning improves in-context learning in language modelscited by 72
  9. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  10. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  11. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  12. 2022Boosting Natural Language Generation from Instructions with Meta-Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 22
  2. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5
  3. 2025Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Onescited by 0
  4. 2026Model of Models: When Does Emitting a Specialist Beat Attending, Adapting, or Tuning?cited by 0