# Design and Evaluation of a Simple Data Interface for Efficient Data Transfer Across Diverse Storage

arXiv: https://arxiv.org/abs/2009.03190 · HTML: https://mlsanity.com/paper/2009.03190/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 1 of 1)

- [Bridging Data Center AI Systems with Edge Computing for Actionable Information Retrieval](https://mlsanity.com/paper/2105.13967/) (2021)

## Abstract (arXiv)

Modern science and engineering computing environments often feature storage systems of different types, from parallel file systems in high-performance computing centers to object stores operated by cloud providers. To enable easy, reliable, secure, and performant data exchange among these different systems, we propose Connector, a pluggable data access architecture for diverse, distributed storage. By abstracting low-level storage system details, this abstraction permits a managed data transfer service (Globus in our case) to interact with a large and easily extended set of storage systems. Equally important, it supports third-party transfers: that is, direct data transfers from source to destination that are initiated by a third-party client but do not engage that third party in the data path. The abstraction also enables management of transfers for performance optimization, error handling, and end-to-end integrity. We present the Connector design, describe implementations for different storage services, evaluate tradeoffs inherent in managed vs. direct transfers, motivate recommended deployment options, and propose a performance model-based method that allows for easy characterization of performance in different contexts without exhaustive benchmarking.