mlsanityData to 14 Oct 2026

Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning

arXiv 2311.03748 ↗First posted 7 Nov 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up the opportunity to make the maximum utilization of large language model knowledge toward structured prediction. Unfortunately, this requires formatting them into specialized augmented format unknown to the base pretrained language model (PLMs) necessitating finetuning to the target format. This significantly bounds its usefulness in data-limited settings where finetuning large models cannot properly generalize to the target format. To address this challenge and leverage PLM knowledge effectively, we propose FISH-DIP, a sample-aware dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples, during the fine-tuning process. By leveraging the dynamism of sparsity, our approach mitigates the impact of well-learned samples and prioritizes underperforming instances for improvement in generalization. Across five tasks of sequence labeling, we demonstrate that FISH-DIP can smoothly optimize the model in low resource settings offering upto 40% performance improvements over full fine-tuning depending on target evaluation settings. Also, compared to in-context learning and other parameter-efficient fine-tuning approaches, FISH-DIP performs comparably or better, notably in extreme low-resource settings.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

30 of 42 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2016Matching Networks for One Shot Learningcited by 2,255
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  11. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 262
  12. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
18 more
  1. 2022Unified Structure Generation for Universal Information Extractioncited by 221
  2. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  3. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  4. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  5. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  6. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 94
  7. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  8. 2021Dialogue State Tracking with a Language Model using Schema-Driven Promptingcited by 94
  9. 2019Induction Networks for Few-Shot Text Classificationcited by 64
  10. 2021CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learningcited by 64
  11. 2019Few-shot Text Classification with Distributional Signaturescited by 58
  12. 2022Autoregressive Structured Prediction with Language Modelscited by 43
  13. 2020Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphscited by 39
  14. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29
  15. 2021Exploring Task Difficulty for Few-Shot Relation Extractioncited by 24
  16. 2021Learning from Miscellaneous Other-Class Words for Few-shot Named Entity Recognitioncited by 21
  17. 2022Few-shot Named Entity Recognition with Entity-level Prototypical Network Enhanced by Dispersedly Distributed Prototypescited by 17
  18. 2020Improving Limited Labeled Dialogue State Tracking with Self-Supervisioncited by 16

Built on by

Cited by year

Citing papers per year: 2024: 11; 2025: 10.

21 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024Large Language Models Meet NLP: A Surveycited by 103
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  5. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  6. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  7. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 14
  8. 2024Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passescited by 10
  9. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  10. 2025Distribution-Aligned Decoding for Efficient LLM Task Adaptationcited by 7
  11. 2025Unbiased Gradient Low-Rank Projectioncited by 7
  12. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
9 more
  1. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  2. 2025Memory-Efficient Fine-Tuning of Transformers via Token Selectioncited by 4
  3. 2024Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Modelscited by 3
  4. 2024Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Modelscited by 2
  5. 2025Toward Realistic Camouflaged Object Detection: Benchmarks and Methodcited by 2
  6. 2025qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMscited by 2
  7. 2025Easy Adaptation: An Efficient Task-Specific Knowledge Injection Method for Large Models in Resource-Constrained Environmentscited by 1
  8. 2025Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanationscited by 0
  9. 2025GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptationcited by 0