Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

This is a Preprint and has not been peer reviewed. This is version 1 of this Preprint.

Downloads

Download Preprint

Authors

Haiying Xu, Kevin Paul, Anderson Banihirwe

Abstract

Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems.
At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform.
Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage.
We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation.
We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.

DOI

https://doi.org/10.31223/X5ZW2T

Subjects

Computer Sciences, Earth Sciences, Physical Sciences and Mathematics

Keywords

HPC, Pangeo benchmark, cloud, HPC, object store, throughput, I/O, Pangeo, Benchmark, cloud, object store, throughput, IO

Dates

Published: 2020-10-21 15:06

Last Updated: 2020-10-21 22:05

License

CC0 1.0 Universal - Public Domain Dedication

Additional Metadata

Data Availability (Reason not available):
Yes

Add a Comment

You must log in to post a comment.


Comments

There are no comments or no comments have been made public for this article.