# An Information-Theoretic Approach to Analyze NLP Classification Tasks

arXiv: https://arxiv.org/abs/2402.00978 · HTML: https://mlsanity.com/paper/2402.00978/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 71 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Neural Machine Translation: A Review and Survey](https://mlsanity.com/paper/1912.02047/) (2019)
- [Evaluating Large Language Models on Controlled Generation Tasks](https://mlsanity.com/paper/2310.14542/) (2023)
- [Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets](https://mlsanity.com/paper/1911.09241/) (2019)
- [A Survey on Machine Reading Comprehension Systems](https://mlsanity.com/paper/2001.01582/) (2020)
- [MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization](https://mlsanity.com/paper/2301.12307/) (2023)
- [Generating Distractors for Reading Comprehension Questions from Real Examinations](https://mlsanity.com/paper/1809.02768/) (2018)
- [Multiple-Choice Question Generation: Towards an Automated Assessment Framework](https://mlsanity.com/paper/2209.11830/) (2022)
- [A quantitative study of NLP approaches to question difficulty estimation](https://mlsanity.com/paper/2305.10236/) (2023)
- [What Makes Reading Comprehension Questions Difficult?](https://mlsanity.com/paper/2203.06342/) (2022)
- [Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution](https://mlsanity.com/paper/2306.13047/) (2023)
- [Analyzing Multiple-Choice Reading and Listening Comprehension Tests](https://mlsanity.com/paper/2307.01076/) (2023)

## Built on by (most cited first, 1 of 1)

- [Question Difficulty Ranking for Multiple-Choice Reading Comprehension](https://mlsanity.com/paper/2404.10704/) (2024)

## Abstract (arXiv)

Understanding the importance of the inputs on the output is useful across many tasks. This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks. Natural language processing (NLP) tasks take either a single element input or multiple element inputs to predict an output variable, where an element is a block of text. Each text element has two components: an associated semantic meaning and a linguistic realization. Multiple-choice reading comprehension (MCRC) and sentiment classification (SC) are selected to showcase the framework. For MCRC, it is found that the context influence on the output compared to the question influence reduces on more challenging datasets. In particular, more challenging contexts allow a greater variation in complexity of questions. Hence, test creators need to carefully consider the choice of the context when designing multiple-choice questions for assessment. For SC, it is found the semantic meaning of the input text dominates (above 80% for all datasets considered) compared to its linguistic realisation when determining the sentiment. The framework is made available at: https://github.com/WangLuran/nlp-element-influence