# Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA

arXiv: https://arxiv.org/abs/2210.05197 · HTML: https://mlsanity.com/paper/2210.05197/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 57 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)

## Built on by (most cited first, 17 of 17)

- [Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation](https://mlsanity.com/paper/2212.08632/) (2022)
- [Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval](https://mlsanity.com/paper/2404.09889/) (2024)
- [Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data](https://mlsanity.com/paper/2402.12869/) (2024)
- [Chain-of-Skills: A Configurable Model for Open-domain Question Answering](https://mlsanity.com/paper/2305.03130/) (2023)
- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directions](https://mlsanity.com/paper/2212.13465/) (2022)
- [cTBLS: Augmenting Large Language Models with Conversational Tables](https://mlsanity.com/paper/2303.12024/) (2023)
- [TTQA-RS- A break-down prompting approach for Multi-hop Table-Text Question Answering with Reasoning and Summarization](https://mlsanity.com/paper/2406.14732/) (2024)
- [Contextualized Data-Wrangling Code Generation in Computational Notebooks](https://mlsanity.com/paper/2409.13551/) (2024)
- [Enhancing Open-Domain Table Question Answering via Syntax- and Structure-aware Dense Retrieval](https://mlsanity.com/paper/2309.10506/) (2023)
- [CSR-RAG: An Efficient Retrieval System for Text-to-SQL on the Enterprise Scale](https://mlsanity.com/paper/2601.06564/) (2026)
- [Denoising Table-Text Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2403.17611/) (2024)
- [HELIOS: Harmonizing Early Fusion, Late Fusion, and LLM Reasoning for Multi-Granular Table-Text Retrieval](https://mlsanity.com/paper/2603.02248/) (2026)
- [Question Answering with Texts and Tables through Deep Reinforcement Learning](https://mlsanity.com/paper/2407.04858/) (2024)
- [Benchmarking Table Comprehension In The Wild](https://mlsanity.com/paper/2412.09884/) (2024)
- [SAGE: Structure Aware Graph Expansion for Retrieval of Heterogeneous Data](https://mlsanity.com/paper/2602.16964/) (2026)
- [Discovery-Driven Integration of Disjoint Tables via Text](https://mlsanity.com/paper/2609.26658/) (2026)

## Abstract (arXiv)

Retrieving evidences from tabular and textual resources is essential for open-domain question answering (OpenQA), which provides more comprehensive information. However, training an effective dense table-text retriever is difficult due to the challenges of table-text discrepancy and data sparsity problem. To address the above challenges, we introduce an optimized OpenQA Table-Text Retriever (OTTeR) to jointly retrieve tabular and textual evidences. Firstly, we propose to enhance mixed-modality representation learning via two mechanisms: modality-enhanced representation and mixed-modality negative sampling strategy. Secondly, to alleviate data sparsity problem and enhance the general retrieval ability, we conduct retrieval-centric mixed-modality synthetic pre-training. Experimental results demonstrate that OTTeR substantially improves the performance of table-and-text retrieval on the OTT-QA dataset. Comprehensive analyses examine the effectiveness of all the proposed mechanisms. Besides, equipped with OTTeR, our OpenQA system achieves the state-of-the-art result on the downstream QA task, with 10.1% absolute improvement in terms of the exact match over the previous best system. All the code and data are available at https://github.com/Jun-jie-Huang/OTTeR.