# Semantic sentence similarity: size does not always matter

arXiv: https://arxiv.org/abs/2106.08648 · HTML: https://mlsanity.com/paper/2106.08648/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Learning Semantic Textual Similarity from Conversations](https://mlsanity.com/paper/1804.07754/) (2018)
- [Learning Visually Grounded Sentence Representations](https://mlsanity.com/paper/1707.06320/) (2017)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech](https://mlsanity.com/paper/1909.08491/) (2019)
- [Learning semantic sentence representations from visually grounded language without lexical knowledge](https://mlsanity.com/paper/1903.11393/) (2019)

## Built on by (most cited first, 5 of 5)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge](https://mlsanity.com/paper/2202.10292/) (2022)
- [Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings](https://mlsanity.com/paper/2210.12857/) (2022)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)
- [An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions](https://mlsanity.com/paper/2410.23955/) (2024)

## Abstract (arXiv)

This study addresses the question whether visually grounded speech recognition (VGS) models learn to capture sentence semantics without access to any prior linguistic knowledge. We produce synthetic and natural spoken versions of a well known semantic textual similarity database and show that our VGS model produces embeddings that correlate well with human semantic similarity judgements. Our results show that a model trained on a small image-caption database outperforms two models trained on much larger databases, indicating that database size is not all that matters. We also investigate the importance of having multiple captions per image and find that this is indeed helpful even if the total number of images is lower, suggesting that paraphrasing is a valuable learning signal. While the general trend in the field is to create ever larger datasets to train models on, our findings indicate other characteristics of the database can just as important important.