A Study of Parallel Data Compression Using Proper Orthogonal Decomposition on the K Computer

Abstract

The growing power of supercomputers continues to improve scientists' ability to model larger, more sophisticated problems in science with higher accuracy. An equally important ability is to make full use of the data output from the simulations to help clarify the modeled phenomena and facilitate the discovery of new phenomena. However, along with the scale of computation, the size of the resulting data has exploded; it becomes infeasible to output most of the data, which defeats the purpose of conducting large-scale simulations. In order to address this issue so that more data may be archived and studied, we have developed a scalable parallel data compression solution to reduce the size of large-scale data with low computational cost and minimal error. We use the proper orthogonal decomposition (POD) method to compress data because this method can effectively extract the main features from the data, and the resulting compressed data can be decompressed in linear time. Our implementation achieves high parallel efficiency with a binary load-distributed approach, which is similar to the binary-swap image composition method. This approach allows us to effectively use all of the processors and to reduce the interprocessor communication cost throughout the parallel compression calculations. The results of tests using the K computer indicate the superior performance of our design and implementation