# IndicBART: A Pre-trained Model for Indic Natural Language Generation

arXiv: https://arxiv.org/abs/2109.02903 · HTML: https://mlsanity.com/paper/2109.02903/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation](https://mlsanity.com/paper/1708.09803/) (2017)
- [PMIndia -- A Collection of Parallel Corpora of Languages of India](https://mlsanity.com/paper/2001.09907/) (2020)
- [Recurrent Stacking of Layers for Compact Neural Machine Translation Models](https://mlsanity.com/paper/1807.05353/) (2018)

## Built on by (most cited first, 30 of 39)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages](https://mlsanity.com/paper/2203.05437/) (2022)
- [BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla](https://mlsanity.com/paper/2205.11081/) (2022)
- [RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization](https://mlsanity.com/paper/2401.14280/) (2024)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [IndicXNLI: Evaluating Multilingual Inference for Indian Languages](https://mlsanity.com/paper/2204.08776/) (2022)
- [PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in India](https://mlsanity.com/paper/2305.08828/) (2023)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [V=arta: A Large-Scale Headline-Generation Dataset for Indic Languages](https://mlsanity.com/paper/2305.05858/) (2023)
- [The State of Large Language Models for African Languages: Progress and Challenges](https://mlsanity.com/paper/2506.02280/) (2025)
- [Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs](https://mlsanity.com/paper/2407.00416/) (2024)
- [Table Question Answering for Low-resourced Indic Languages](https://mlsanity.com/paper/2410.03576/) (2024)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2304.09388/) (2023)
- [Mukhyansh: A Headline Generation Dataset for Indic Languages](https://mlsanity.com/paper/2311.17743/) (2023)
- [IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages](https://mlsanity.com/paper/2312.09508/) (2023)
- [Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models](https://mlsanity.com/paper/2305.13085/) (2023)
- [S=amayik: A Benchmark and Dataset for English-Sanskrit Translation](https://mlsanity.com/paper/2305.14004/) (2023)
- [RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languages](https://mlsanity.com/paper/2412.15248/) (2024)
- [Summarizing Indian Languages using Multilingual Transformers based Models](https://mlsanity.com/paper/2303.16657/) (2023)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [The Less the Merrier? Investigating Language Representation in Multilingual Models](https://mlsanity.com/paper/2310.13228/) (2023)
- [AgriLLM: Harnessing Transformers for Farmer Queries](https://mlsanity.com/paper/2407.04721/) (2024)
- [BanglaByT5: Byte-Level Modelling for Bangla](https://mlsanity.com/paper/2505.17102/) (2025)
- [Implementing Deep Learning-Based Approaches for Article Summarization in Indian Languages](https://mlsanity.com/paper/2212.05702/) (2022)
- [Evaluating Inter-Bilingual Semantic Parsing for Indian Languages](https://mlsanity.com/paper/2304.13005/) (2023)
- [Multilingual Bias Detection and Mitigation for Indian Languages](https://mlsanity.com/paper/2312.15181/) (2023)
- [Machine Translation with Large Language Models: Decoder Only vs. Encoder-Decoder](https://mlsanity.com/paper/2409.13747/) (2024)

## Abstract (arXiv)

In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.