Technical note: A procedure to clean, decompose and aggregate time series

This is a Preprint and has not been peer reviewed. The published version of this Preprint is available: https://doi.org/10.5194/hess-27-349-2023. This is version 2 of this Preprint.

Add a Comment

You must log in to post a comment.


Comments

There are no comments or no comments have been made public for this article.

Downloads

Download Preprint

Authors

François Ritter

Abstract

Errors, gaps and outliers complicate and sometimes invalidate the analysis of time series. While most fields have developed their own strategy to clean the raw data, no generic procedure has been promoted to standardize the pre-processing. This lack of harmonization makes the inter-comparison of studies difficult, and leads to screening methods that can be arbitrary or case-specific. This study provides a generic pre-processing procedure implemented in R (ctbi, for cyclic/trend decomposition using bin interpolation) dedicated to univariate time series. Ctbi is based on data binning and decomposes the time series into a long-term trend and a cyclic component (quantified by a new metric, the Stacked Cycles Index) to finally aggregate the data. Outliers are flagged with an enhanced boxplot rule called Logbox that corrects biases due to the sample size and that is adapted to non-Gaussian residuals. Three different Earth Science datasets (contaminated with gaps and outliers) are successfully cleaned and aggregated with ctbi. This illustrates the robustness of this procedure that can be valuable to any discipline.

DOI

https://doi.org/10.31223/X5107C

Subjects

Computer Sciences, Earth Sciences, Environmental Sciences, Physical Sciences and Mathematics, Statistics and Probability

Keywords

Outliers, time series, pre-processing, aggregation

Dates

Published: 2022-11-17 04:35

Last Updated: 2023-01-20 06:21

Older Versions
License

CC BY Attribution 4.0 International

Additional Metadata

Conflict of interest statement:
None

Data Availability (Reason not available):
https://github.com/fritte2/ctbi_article