# Test-Time Self-Adaptive Small Language Models for Question Answering

arXiv: https://arxiv.org/abs/2310.13307 · HTML: https://mlsanity.com/paper/2310.13307/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems](https://mlsanity.com/paper/2010.06028/) (2020)
- [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14106/) (2023)
- [Efficient Test Time Adapter Ensembling for Low-resource Language Varieties](https://mlsanity.com/paper/2109.04877/) (2021)
- [Self-Supervised Test-Time Learning for Reading Comprehension](https://mlsanity.com/paper/2103.11263/) (2021)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)
- [Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker](https://mlsanity.com/paper/2305.13729/) (2023)
- [Evaluating the Robustness of Discrete Prompts](https://mlsanity.com/paper/2302.05619/) (2023)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [ZARA: Improving Few-Shot Self-Rationalization for Small Language Models](https://mlsanity.com/paper/2305.07355/) (2023)

## Built on by (most cited first, 7 of 7)

- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness](https://mlsanity.com/paper/2411.03350/) (2024)
- [UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation](https://mlsanity.com/paper/2405.01022/) (2024)
- [CTTA-T: Continual Test-Time Adaptation for Text Understanding via Teacher-Student with a Domain-aware and Generalized Teacher](https://mlsanity.com/paper/2512.18321/) (2025)
- [IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation](https://mlsanity.com/paper/2510.20377/) (2025)
- [A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models](https://mlsanity.com/paper/2604.04168/) (2026)
- [EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering](https://mlsanity.com/paper/2606.06906/) (2026)

## Abstract (arXiv)

Recent instruction-finetuned large language models (LMs) have achieved notable performances in various tasks, such as question-answering (QA). However, despite their ability to memorize a vast amount of general knowledge across diverse tasks, they might be suboptimal on specific tasks due to their limited capacity to transfer and adapt knowledge to target tasks. Moreover, further finetuning LMs with labeled datasets is often infeasible due to their absence, but it is also questionable if we can transfer smaller LMs having limited knowledge only with unlabeled test data. In this work, we show and investigate the capabilities of smaller self-adaptive LMs, only with unlabeled test data. In particular, we first stochastically generate multiple answers, and then ensemble them while filtering out low-quality samples to mitigate noise from inaccurate labels. Our proposed self-adaption strategy demonstrates significant performance improvements on benchmark QA datasets with higher robustness across diverse prompts, enabling LMs to stay stable. Code is available at: https://github.com/starsuzi/T-SAS.