mlsanityData to 7 Oct 2026

Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models

arXiv 2411.00686 ↗First posted 1 Nov 2024cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers. This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. Additionally, combining LaPael with data-level paraphrasing further enhances performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
34 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2023Textbooks Are All You Needcited by 561
  8. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  9. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  13. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  14. 2017Concrete Dropoutcited by 227
  15. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  16. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  17. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  18. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  20. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  21. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  22. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  23. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  24. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  25. 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 94
  26. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  27. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  28. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  29. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  30. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  31. 2021ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collectionscited by 30
  32. 2021Learning to Perturb Word Embeddings for Out-of-distribution QAcited by 13
  33. 2020MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architecturescited by 7
  34. 2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generationcited by 0
  2. 2026KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Trainingcited by 0
  3. 2026Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agentscited by 0