mlsanityData to 14 Oct 2026

Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection

arXiv 2205.10455 ↗First posted 20 May 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents. In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets. Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document. Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

22 of 39 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  11. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  12. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
10 more
  1. 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
  2. 2020A Paragraph-level Multi-task Learning Model for Scientific Fact-Verificationcited by 29
  3. 2016A Compare-Aggregate Model for Matching Text Sequencescited by 23
  4. 2020The Cascade Transformer: an Application for Efficient Answer Sentence Selectioncited by 20
  5. 2019A Compare-Aggregate Model with Latent Clustering for Answer Selectioncited by 19
  6. 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
  7. 2020Toward Better Storylines with Sentence-Level Language Modelscited by 14
  8. 2021Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inferencecited by 13
  9. 2019Language Model Pre-training for Hierarchical Document Representationscited by 7
  10. 2022Paragraph-based Transformer Pre-training for Multi-Sentence Inferencecited by 7

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 4; 2024: 2.

9 papers.

  1. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  2. 2022Knowledge Transfer from Answer Ranking to Answer Generationcited by 6
  3. 2022Effective Pre-Training Objectives for Transformer-based Autoencoderscited by 3
  4. 2023Learning Answer Generation using Supervision from Automatic Question Answering Evaluatorscited by 3
  5. 2023Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languagescited by 2
  6. 2023Context-Aware Transformer Pre-Training for Answer Sentence Selectioncited by 1
  7. 2023SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative Referencescited by 1
  8. 2024Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMscited by 1
  9. 2024Domain Adaptation of Multilingual Semantic Search -- Literature Reviewcited by 0