# Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information

arXiv: https://arxiv.org/abs/2109.13073 · HTML: https://mlsanity.com/paper/2109.13073/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 61 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Neural Summarization by Extracting Sentences and Words](https://mlsanity.com/paper/1603.07252/) (2016)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [Neural Document Summarization by Jointly Learning to Score and Select Sentences](https://mlsanity.com/paper/1807.02305/) (2018)
- [Applying CodeBERT for Automated Program Repair of Java Simple Bugs](https://mlsanity.com/paper/2103.11626/) (2021)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [A Multi-Modal Transformer-based Code Summarization Approach for Smart Contracts](https://mlsanity.com/paper/2103.07164/) (2021)
- [Generating Question Titles for Stack Overflow from Mined Code Snippets](https://mlsanity.com/paper/2005.10157/) (2020)

## Built on by (most cited first, 5 of 5)

- [Exploring and Unleashing the Power of Large Language Models in Automated Code Translation](https://mlsanity.com/paper/2404.14646/) (2024)
- [From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions](https://mlsanity.com/paper/2402.17633/) (2024)
- [Automating Code-Related Tasks Through Transformers: The Impact of Pre-training](https://mlsanity.com/paper/2302.04048/) (2023)
- [Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer](https://mlsanity.com/paper/2208.11523/) (2022)
- [Learning to Learn to Predict Performance Regressions in Production at Meta](https://mlsanity.com/paper/2208.04351/) (2022)

## Abstract (arXiv)

Context: Stack Overflow is very helpful for software developers who are seeking answers to programming problems. Previous studies have shown that a growing number of questions are of low quality and thus obtain less attention from potential answerers. Gao et al. proposed an LSTM-based model (i.e., BiLSTM-CC) to automatically generate question titles from the code snippets to improve the question quality. However, only using the code snippets in the question body cannot provide sufficient information for title generation, and LSTMs cannot capture the long-range dependencies between tokens. Objective: This paper proposes CCBERT, a deep learning based novel model to enhance the performance of question title generation by making full use of the bi-modal information of the entire question body. Method: CCBERT follows the encoder-decoder paradigm and uses CodeBERT to encode the question body into hidden representations, a stacked Transformer decoder to generate predicted tokens, and an additional copy attention layer to refine the output distribution. Both the encoder and decoder perform the multi-head self-attention operation to better capture the long-range dependencies. This paper builds a dataset containing around 200,000 high-quality questions filtered from the data officially published by Stack Overflow to verify the effectiveness of the CCBERT model. Results: CCBERT outperforms all the baseline models on the dataset. Experiments on both code-only and low-resource datasets show the superiority of CCBERT with less performance degradation. The human evaluation also shows the excellent performance of CCBERT concerning both readability and correlation criteria.