Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

This is a Preprint and has not been peer reviewed. This is version 1 of this Preprint.


Download Preprint


Haiying Xu, Kevin Paul, Anderson Banihirwe


Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems.
At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform.
Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage.
We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation.
We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.



Computer Sciences, Earth Sciences, Physical Sciences and Mathematics


HPC, Pangeo benchmark, cloud, HPC, object store, throughput, I/O, Pangeo, Benchmark, cloud, object store, throughput, IO


Published: 2020-10-21 08:06

Last Updated: 2020-10-21 15:05


CC0 1.0 Universal - Public Domain Dedication

Additional Metadata

Data Availability (Reason not available):

Add a Comment

You must log in to post a comment.


There are no comments or no comments have been made public for this article.