# Cross-Domain Evaluation of a Deep Learning-Based Type Inference System

arXiv: https://arxiv.org/abs/2208.09189 · HTML: https://mlsanity.com/paper/2208.09189/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 19 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [LambdaNet: Probabilistic Type Inference using Graph Neural Networks](https://mlsanity.com/paper/2005.02161/) (2020)
- [TypeWriter: Neural Type Prediction with Search-based Validation](https://mlsanity.com/paper/1912.03768/) (2019)
- [Typilus: Neural Type Hints](https://mlsanity.com/paper/2004.10657/) (2020)
- [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018)
- [Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python](https://mlsanity.com/paper/2101.04470/) (2021)
- [OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints](https://mlsanity.com/paper/2004.00348/) (2020)
- [ManyTypes4Py: A Benchmark Python Dataset for Machine Learning-based Type Inference](https://mlsanity.com/paper/2104.04706/) (2021)
- [PYInfer: Deep Learning Semantic Type Inference for Python Variables](https://mlsanity.com/paper/2106.14316/) (2021)
- [DLTPy: Deep Learning Type Inference of Python Function Signatures using Natural Language Context](https://mlsanity.com/paper/1912.00680/) (2019)

## Built on by (most cited first, 1 of 1)

- [TypeT5: Seq2seq Type Inference using Static Analysis](https://mlsanity.com/paper/2303.09564/) (2023)

## Abstract (arXiv)

Optional type annotations allow for enriching dynamic programming languages with static typing features like better Integrated Development Environment (IDE) support, more precise program analysis, and early detection and prevention of type-related runtime errors. Machine learning-based type inference promises interesting results for automating this task. However, the practical usage of such systems depends on their ability to generalize across different domains, as they are often applied outside their training domain. In this work, we investigate Type4Py as a representative of state-of-the-art deep learning-based type inference systems, by conducting extensive cross-domain experiments. Thereby, we address the following problems: class imbalances, out-of-vocabulary words, dataset shifts, and unknown classes. To perform such experiments, we use the datasets ManyTypes4Py and CrossDomainTypes4Py. The latter we introduce in this paper. Our dataset enables the evaluation of type inference systems in different domains of software projects and has over 1,000,000 type annotations mined on the platforms GitHub and Libraries. It consists of data from the two domains web development and scientific calculation. Through our experiments, we detect that the shifts in the dataset and the long-tailed distribution with many rare and unknown data types decrease the performance of the deep learning-based type inference system drastically. In this context, we test unsupervised domain adaptation methods and fine-tuning to overcome these issues. Moreover, we investigate the impact of out-of-vocabulary words.