mlsanityData to 14 Oct 2026

PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning

arXiv 2407.02211 ↗First posted 2 Jul 2024cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent advances in fine-tuning large language models (LLMs) have greatly enhanced their usage in domain-specific tasks. Despite the success, fine-tuning continues to rely on repeated and lengthy prompts, which escalate computational expenses, require more resources, and lead to slower inference. In this paper, we present a novel approach, PromptIntern, which internalizes prompt knowledge during model fine-tuning to achieve efficient inference and save costs. Instead of compressing the prompts for a vanilla model, PromptIntern aims to embed the recurrent prompt directly into the model parameters. We design a fine-tuning pipeline that includes instruction template compression, few-shot example absorption, and a progressive internalization strategy, effectively diminishing the need for intricate prompts during inference. Comprehensive experiments on challenging NL2Code tasks demonstrate that our method reduces input tokens by more than 90%, accelerates inference by 4.2 times, and reduces monetary inference costs by 88.3%.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2017Prototypical Networks for Few-shot Learningcited by 3,015
22 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  5. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  10. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  11. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  13. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  14. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  15. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  16. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  17. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 85
  18. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  19. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 41
  20. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  21. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoningcited by 32
  22. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  2. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  3. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  4. 2024PageRank Bandits for Link Predictioncited by 14
  5. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  6. 2024Generative Prompt Internalizationcited by 8
  7. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  8. 2024STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsingcited by 3
  9. 2025Enterprise AI Must Enforce Participant-Aware Access Controlcited by 3
  10. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  11. 2025Analyzing and Internalizing Complex Policy Documents for LLM Agentscited by 2
  12. 2026SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalizationcited by 1
1 more
  1. 2024Rephrase and Contrast: Fine-Tuning Language Models for Enhanced Understanding of Communication and Computer Networkscited by 0