Dynamic I/O Budget Reallocation For In Situ Wavelet Compression

Abstract

In situ wavelet compression is a potential solution for enabling post hoc visualization on supercomputers with slow I/O systems. While this in situ compression is typically accomplished by allocating an equal storage budget to each parallel process, we propose an adaptive approach. With our approach, we introduce an assessment step prior to compression, where each process characterizes the variation in its portion of the data, and then dynamically adapts storage budgets to the processes with the most variation. We conducted experiments comparing our adaptive approach with the traditional, non-adaptive approach, on two different simulation codes with concurrencies of 512 cores and mesh resolutions of one billion cells. Our findings show that our adaptive approach yields three orders of magnitude of improvement for one simulation and is not harmful for the other.