Understanding the Performance of Ceph Block Storage for Hyper-Converged Cloud with All Flash Storage

Hyper-converged cloud refers to an architecture that an operator runs compute and storage services on the same set of physical servers. Although the hyper-converged design comes with a number of benefits, it makes crucial operational tasks, such as capacity planning and cost analysis, fairly complicated. The problem becomes more onerous if we consider a complex distributed system, such as Ceph, for the cloud with the proliferation of SSD drives. In this paper, we aim to answer some of these questions based on comprehensive microbenchmarks, and consequently better understand the behavior of Ceph in a hyper-converged cloud with all-flash storage. We reported our findings based on the study, devised a cost model and compared the cost of hyper-converged architecture with dedicated storage architecture. Additionally we summarized our experience based on the interactions with many teams at AT&T in the past couple of years.