mlsanityData to 14 Oct 2026

InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding

arXiv 2306.04933 ↗First posted 8 Jun 2023cs.CL cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Soft prompt tuning achieves superior performances across a wide range of few-shot tasks. However, the performances of prompt tuning can be highly sensitive to the initialization of the prompts. We also empirically observe that conventional prompt tuning methods cannot encode and learn sufficient task-relevant information from prompt tokens. In this work, we develop an information-theoretic framework that formulates soft prompt tuning as maximizing mutual information between prompts and other model parameters (or encoded representations). This novel view helps us to develop a more efficient, accurate and robust soft prompt tuning method InfoPrompt. With this framework, we develop two novel mutual information based loss functions, to (i) discover proper prompt initialization for the downstream tasks and learn sufficient task-relevant information from prompt tokens and (ii) encourage the output representation from the pretrained language model to be more aware of the task-relevant information captured in the learnt prompt. Extensive experiments validate that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods. Finally, we provide a formal theoretical result for showing to show that gradient descent type algorithm can be used to train our mutual information loss.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 76 of its 112 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

76 of 112 references.

References · 76

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
38 more
  1. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  2. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  3. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  4. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  5. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  6. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  7. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  8. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  9. 2023Representational Strengths and Limitations of Transformerscited by 133
  10. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  11. 2023Fast Attention Requires Bounded Entriescited by 102
  12. 2018Solving Linear Programs in the Current Matrix Multiplication Timecited by 100
  13. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  14. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  15. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  16. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  17. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  18. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  19. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  20. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  21. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  22. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  23. 2020A Refined Laser Method and Faster Matrix Multiplicationcited by 56
  24. 2019Solving Empirical Risk Minimization in the Current Matrix Multiplication Timecited by 53
  25. 2020A Faster Interior Point Method for Semidefinite Programmingcited by 53
  26. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  27. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  28. 2016Joint Extraction of Events and Entities within a Document Contextcited by 48
  29. 2023An Over-parameterized Exponential Regressioncited by 45
  30. 2020Faster Dynamic Matrix Inverse for Faster LPscited by 44
  31. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  32. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  33. 2019A Deterministic Linear Program Solver in Current Matrix Multiplication Timecited by 42
  34. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  35. 2019BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principlecited by 39
  36. 2023Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Modelscited by 37
  37. 2023Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimensioncited by 37
  38. 2022A Faster Small Treewidth SDP Solvercited by 36

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 8; 2025: 7; 2026: 7.

30 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 50
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  5. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  6. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  7. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  8. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  9. 2023Beyond Classical Attention: Quantum Attention for Scalable Computationcited by 18
  10. 2023Zero-th Order Algorithm for Softmax Attention Optimizationcited by 16
  11. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  12. 2023A Unified Scheme of ResNet and Softmaxcited by 12
18 more
  1. 2023Efficient SGD Neural Network Training via Sublinear Activated Neuron Identificationcited by 11
  2. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  3. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  4. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  5. 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agentscited by 7
  6. 2024Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RLcited by 6
  7. 2024Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learningcited by 4
  8. 2024DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendationcited by 3
  9. 2026SoftSkill: Behavioral Compression for Contextual Adaptationcited by 3
  10. 2026Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Promptcited by 2
  11. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  12. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  13. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 1
  14. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  15. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  16. 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Drivingcited by 1
  17. 2025Information-Theoretic Complementary Prompts for Improved Continual Text Classificationcited by 0
  18. 2026Cancerous Backdoor Attack: Gradual Activation via Invisible Tokens and A Stage Controllercited by 0