# Adaptive Prompt Learning with Distilled Connective Knowledge for Implicit Discourse Relation Recognition

arXiv: https://arxiv.org/abs/2309.07561 · HTML: https://mlsanity.com/paper/2309.07561/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [Knowledge Distillation via Route Constrained Optimization](https://mlsanity.com/paper/1904.09149/) (2019)
- [Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System](https://mlsanity.com/paper/1910.08381/) (2019)
- [On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification](https://mlsanity.com/paper/2004.12617/) (2020)
- [Conditional Teacher-Student Learning](https://mlsanity.com/paper/1904.12399/) (2019)
- [Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification](https://mlsanity.com/paper/1804.10069/) (2018)
- [Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification](https://mlsanity.com/paper/1704.00217/) (2017)
- [Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention](https://mlsanity.com/paper/1609.06380/) (2016)
- [Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation](https://mlsanity.com/paper/2306.06480/) (2023)
- [A Survey of Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2203.02982/) (2022)
- [LabelPrompt: Effective Prompt-based Learning for Relation Classification](https://mlsanity.com/paper/2302.08068/) (2023)

## Abstract (arXiv)

Implicit discourse relation recognition (IDRR) aims at recognizing the discourse relation between two text segments without an explicit connective. Recently, the prompt learning has just been applied to the IDRR task with great performance improvements over various neural network-based approaches. However, the discrete nature of the state-art-of-art prompting approach requires manual design of templates and answers, a big hurdle for its practical applications. In this paper, we propose a continuous version of prompt learning together with connective knowledge distillation, called AdaptPrompt, to reduce manual design efforts via continuous prompting while further improving performance via knowledge transfer. In particular, we design and train a few virtual tokens to form continuous templates and automatically select the most suitable one by gradient search in the embedding space. We also design an answer-relation mapping rule to generate a few virtual answers as the answer space. Furthermore, we notice the importance of annotated connectives in the training dataset and design a teacher-student architecture for knowledge transfer. Experiments on the up-to-date PDTB Corpus V3.0 validate our design objectives in terms of the better relation recognition performance over the state-of-the-art competitors.