# MQDD: Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain

arXiv: https://arxiv.org/abs/2203.14093 · HTML: https://mlsanity.com/paper/2203.14093/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence](https://mlsanity.com/paper/1903.09588/) (2019)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Learned in Translation: Contextualized Word Vectors](https://mlsanity.com/paper/1708.00107/) (2017)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019)
- [ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking](https://mlsanity.com/paper/1912.02972/) (2019)
- [NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging](https://mlsanity.com/paper/2112.00405/) (2021)
- [A Literature Study of Embeddings on Source Code](https://mlsanity.com/paper/1904.03061/) (2019)
- [Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent](https://mlsanity.com/paper/2008.12193/) (2020)
- [Code Search based on Context-aware Code Translation](https://mlsanity.com/paper/2202.08029/) (2022)

## Abstract (arXiv)

This work proposes a new pipeline for leveraging data collected on the Stack Overflow website for pre-training a multimodal model for searching duplicates on question answering websites. Our multimodal model is trained on question descriptions and source codes in multiple programming languages. We design two new learning objectives to improve duplicate detection capabilities. The result of this work is a mature, fine-tuned Multimodal Question Duplicity Detection (MQDD) model, ready to be integrated into a Stack Overflow search system, where it can help users find answers for already answered questions. Alongside the MQDD model, we release two datasets related to the software engineering domain. The first Stack Overflow Dataset (SOD) represents a massive corpus of paired questions and answers. The second Stack Overflow Duplicity Dataset (SODD) contains data for training duplicate detection models.