# BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer

arXiv: https://arxiv.org/abs/2304.09649 · HTML: https://mlsanity.com/paper/2304.09649/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer](https://mlsanity.com/paper/2212.02027/) (2022)
- [A Fine-Grained Sentiment Dataset for Norwegian](https://mlsanity.com/paper/1911.12722/) (2019)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [NorNE: Annotating Named Entities for Norwegian](https://mlsanity.com/paper/1911.12146/) (2019)
- [NoReC: The Norwegian Review Corpus](https://mlsanity.com/paper/1710.05370/) (2017)

## Built on by (most cited first, 2 of 2)

- [Manipulating Large Language Models to Increase Product Visibility](https://mlsanity.com/paper/2404.07981/) (2024)
- [More Room for Language: Investigating the Effect of Retrieval on Language Models](https://mlsanity.com/paper/2404.10939/) (2024)

## Abstract (arXiv)

Retrieval-based language models are increasingly employed in question-answering tasks. These models search in a corpus of documents for relevant information instead of having all factual knowledge stored in its parameters, thereby enhancing efficiency, transparency, and adaptability. We develop the first Norwegian retrieval-based model by adapting the REALM framework and evaluating it on various tasks. After training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks. Results show that retrieval augmented language modeling improves the reader's performance on extractive question-answering, suggesting that this type of training improves language models' general ability to use context and that this does not happen at the expense of other abilities such as part-of-speech tagging, dependency parsing, named entity recognition, and lemmatization. Code, trained models, and data are made publicly available.