mlsanityData to 7 Oct 2026

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

arXiv 2204.11989 ↗First posted 25 Apr 2022cs.IR cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have improved effectiveness on numerous tasks, including ad-hoc retrieval. Recent work has shown that continuing to pretrain a language model with auxiliary objectives before fine-tuning on the retrieval task can further improve retrieval effectiveness. Unlike monolingual retrieval, designing an appropriate auxiliary task for cross-language mappings is challenging. To address this challenge, we use comparable Wikipedia articles in different languages to further pretrain off-the-shelf multilingual pretrained models before fine-tuning on the retrieval task. We show that our approach yields improvements in retrieval effectiveness.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 25 Apr 2022.

Builds on

29 of 47 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  11. 2019Multi-Stage Document Ranking with BERTcited by 200
  12. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
17 more
  1. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  2. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  3. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
  4. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  5. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  6. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  7. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  8. 2020Cross-lingual Information Retrieval with BERTcited by 27
  9. 2021Simplified Data Wrangling with ir_datasetscited by 27
  10. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  11. 2019Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learningcited by 16
  12. 2020The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020cited by 16
  13. 2020A Study of Neural Matching Models for Cross-lingual IRcited by 12
  14. 2022HC4: A New Suite of Test Collections for Ad Hoc CLIRcited by 10
  15. 2019Cross-Lingual Relevance Transfer for Document Retrievalcited by 9
  16. 2021Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrievalcited by 8
  17. 2021Cross-Lingual Training with Dense Retrieval for Document Retrievalcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 6; 2025: 2.

14 papers.

Sort:

By area:

  1. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  2. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  3. 2023PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessmentcited by 19
  4. 2022Neural Approaches to Multilingual Information Retrievalcited by 18
  5. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  6. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  7. 2025MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusioncited by 8
  8. 2024ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrievalcited by 7
  9. 2023Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checkingcited by 5
  10. 2023Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Datacited by 4
  11. 2022Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapterscited by 3
  12. 2024Distillation for Multilingual Information Retrievalcited by 3
2 more
  1. 2024Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptationcited by 1
  2. 2024Extending Translate-Train for ColBERT-X to African Language CLIRcited by 1