mlsanityData to 14 Oct 2026

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank

arXiv 2410.23066 ↗First posted 30 Oct 2024cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT - Pretrained and Leveraged Attention - a plug-and-play strategy for initializing attention. PLANT works by planting label-specific attention using a pretrained Learning-to-Rank model guided by mutual information gain. This architecture-agnostic approach integrates seamlessly with large language model backbones such as Mistral-7B, LLaMA3-8B, DeepSeek-V3, and Phi-3. PLANT outperforms state-of-the-art methods across tasks including ICD coding, legal topic classification, and content recommendation. Gains are especially pronounced in few-shot settings, with substantial improvements on rare labels. Ablation studies confirm that attention initialization is a key driver of these gains. For code and trained models, see https://github.com/debjyotiSRoy/xcube/tree/plant

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 95 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

51 of 95 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2024Mixtral of Expertscited by 1,823
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2020Asymmetric Loss For Multi-Label Classificationcited by 381
  12. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
38 more
  1. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 234
  2. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  4. 2018AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classificationcited by 102
  5. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  6. 2019ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Networkcited by 73
  7. 2021Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classificationcited by 67
  8. 2022Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Codingcited by 53
  9. 2020PECOS: Prediction for Enormous and Correlated Output Spacescited by 50
  10. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  11. 2022PLM-ICD: Automatic ICD Coding with Pretrained Language Modelscited by 49
  12. 2023Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studycited by 49
  13. 2020A Label Attention Model for ICD Coding from Clinical Textcited by 48
  14. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  15. 2021CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notescited by 20
  16. 2023Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classificationcited by 19
  17. 2024CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learningcited by 18
  18. 2022Multi-label Few-shot ICD Coding as Autoregressive Generation with Promptcited by 17
  19. 2023Transformers in Healthcare: A Surveycited by 17
  20. 2023Automated clinical coding using off-the-shelf large language modelscited by 14
  21. 2024Can GPT-3.5 Generate and Code Discharge Summaries?cited by 13
  22. 2022CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classificationcited by 12
  23. 2024An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Recordscited by 12
  24. 2022ICDBigBird: A Contextual Embedding Model for ICD Code Classificationcited by 10
  25. 2021InceptionXML: A Lightweight Framework with Synchronized Negative Sampling for Short Text Extreme Classificationcited by 8
  26. 2024Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendationcited by 8
  27. 2023Modelling Temporal Document Sequences for Clinical ICD Codingcited by 7
  28. 2023Dual-Encoders for Extreme Multi-Label Classificationcited by 7
  29. 2023Compositional Generalization for Multi-label Text Classification: A Data-Augmentation Approachcited by 7
  30. 2023A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approachescited by 6
  31. 2023Surpassing GPT-4 Medical Coding with a Two-Stage Approachcited by 6
  32. 2023ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classificationcited by 5
  33. 2023MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularitiescited by 4
  34. 2023Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learningcited by 4
  35. 2024DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Predictioncited by 4
  36. 2023MatchXML: An Efficient Text-label Matching Framework for Extreme Multi-label Text Classificationcited by 3
  37. 2024Learning label-label correlations in Extreme Multi-label Classification via Label Featurescited by 3
  38. 2024Auxiliary Knowledge-Induced Learning for Automatic Multi-Label Medical Document Classificationcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.