# SILT: Efficient transformer training for inter-lingual inference arXiv: https://arxiv.org/abs/2103.09635 ยท HTML: https://mlsanity.com/paper/2103.09635/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 43 references that are in the map. - First posted on arXiv 17 Mar 2021. ## Builds on (most cited first, 30 of 37) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017) - [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018) - [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015) - [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019) - [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018) - [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017) - [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020) - [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016) - [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016) - [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020) - [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018) - [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018) - [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019) - [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019) - [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020) - [OCNLI: Original Chinese Natural Language Inference](https://mlsanity.com/paper/2010.05444/) (2020) - [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018) - [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019) - [Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning](https://mlsanity.com/paper/1804.00079/) (2018) - [Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference](https://mlsanity.com/paper/1907.04380/) (2019) - [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017) ## Built on by (most cited first, 5 of 5) - [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024) - [Evaluation Benchmarks for Spanish Sentence Representations](https://mlsanity.com/paper/2204.07571/) (2022) - [Associative Learning Mechanism for Drug-Target Interaction Prediction](https://mlsanity.com/paper/2205.15364/) (2022) - [DisTrack: a new Tool for Semi-automatic Misinformation Tracking in Online Social Networks](https://mlsanity.com/paper/2408.00633/) (2024) - [ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships](https://mlsanity.com/paper/2503.08803/) (2025) ## Abstract (arXiv) The ability of transformers to perform precision tasks such as question answering, Natural Language Inference (NLI) or summarising, have enabled them to be ranked as one of the best paradigm to address Natural Language Processing (NLP) tasks. NLI is one of the best scenarios to test these architectures, due to the knowledge required to understand complex sentences and established relationships between a hypothesis and a premise. Nevertheless, these models suffer from incapacity to generalise to other domains or difficulties to face multilingual and interlingual scenarios. The leading pathway in the literature to address these issues involve designing and training extremely large architectures, which leads to unpredictable behaviours and to establish barriers which impede broad access and fine tuning. In this paper, we propose a new architecture called Siamese Inter-Lingual Transformer (SILT), to efficiently align multilingual embeddings for Natural Language Inference, allowing for unmatched language pairs to be processed. SILT leverages siamese pre-trained multi-lingual transformers with frozen weights where the two input sentences attend each other to later be combined through a matrix alignment method. The experimental results carried out in this paper evidence that SILT allows to reduce drastically the number of trainable parameters while allowing for inter-lingual NLI and achieving state-of-the-art performance on common benchmarks. We make our code and dataset available at https://github.com/jahuerta92/siamese-inter-lingual-transformer.