mlsanityData to 7 Oct 2026

Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data

arXiv 2404.02422 ↗First posted 3 Apr 2024cs.CL cs.LGcited by 7

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 19 of its 26 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

19 of 26 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
7 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  3. 2018Universal Sentence Encodercited by 485
  4. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  5. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  6. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  7. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMscited by 23
  2. 2024Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisionscited by 11
  3. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  4. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictionscited by 9
  5. 2024An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classificationcited by 5
  6. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  7. 2025A Comprehensive Dataset for Human vs. AI Generated Text Detectioncited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Models Are Zero-Shot Text Classifiers
  2. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  3. 2025Large Language Models as Universal Predictors? An Empirical Study on Small Tabular Datasets
  4. 2023Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
  5. 2026Curate-Train-Refine: A Closed-Loop Agentic Framework for Zero Shot Classification
  6. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?
  7. 2023Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM
  8. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models