NPO Tsukuba OSS Technical Support Center (OSS Tsukuba)

Use Case Overview

The Subaru Telescope data analysis case is an excellent example of how Gfarm's distributed storage technology & Pwrake's advanced workflow management can eliminate bottlenecks in I/O-intensive big data processing & maximize computational resource utilization efficiency. In large-scale processing where I/O tends to be a bottleneck, the combination of Gfarm & Pwrake becomes a powerful solution supporting breakthroughs in scientific research.

This case study introduces an initiative that significantly improved processing speed by leveraging Gfarm & Pwrake in the analysis of massive image data from Japan's cutting-edge astronomical observation project, the "Subaru Telescope Hyper Suprime-Cam (HSC)."

Challenge

Massive I/O & Limitations of Existing Systems

The wide-field imaging observations by HSC on the Subaru Telescope constitute a major project aimed at observing distant galaxies & elucidating the distribution of dark matter in the universe. The observation data is extremely large-scale, with raw data from a single night alone reaching 300 GB (approximately 10 times that after processing).

There was a strong need to quickly analyze such massive image data (big data) & see results promptly. Particularly for follow-up observations of sudden astronomical phenomena such as supernovae, speeding up processing time was essential. However, the parallel processing system implemented in the conventional data analysis pipeline (hscPipe) had shortcomings that prevented scale-out (expansion) in terms of data input/output (I/O) performance & efficient use of computational resources (CPU cores). Specifically, even when attempting to use many compute nodes, I/O became a bottleneck, preventing the system from fully utilizing the computer's capabilities.

Solution

Highly Efficient Core Usage & Scalable I/O Performance

In this project, the parallel file system Gfarm & workflow system Pwrake were applied to hscPipe with the aim of improving I/O performance & core utilization rate.

1Scalable I/O Performance

Gfarm has a structure (Node-local Storage) where data is distributed & stored in the local storage of compute nodes, & I/O performance improves proportionally (scales out) according to the number of compute nodes. This eliminated the I/O bottleneck that occurred in conventional file systems.

2Efficient Task Execution

Pwrake schedules tasks to execute processing close to where data is stored, considering file locality (where data is saved). This reduced the phenomenon of CPU cores becoming idle while compute nodes wait for task completion, efficiently improving core utilization rate.

3Achieving Dramatic Speed Improvement

In a comparison experiment using small-scale observation data (1/4 of one night), the system combining Pwrake & Gfarm achieved a 2.2x speedup compared to the conventional hscPipe parallel execution system (reducing processing time from 6521 seconds to 2968 seconds).

4Demonstration of Large-Scale Processing

Even with ultra-large-scale processing of 58 days of observation data (5.3TB input, 46TB output), processing was completed in 52 hours using 30 nodes & 480 cores. At this time, a very high efficiency of 95.4% core utilization rate was achieved, demonstrating that the system can withstand large-scale data processing.

Future Development

Through this technology, we aim to pioneer a new research field called "Statistical Computational Astrophysics" that fuses large-scale astronomical big data with information statistics. Faster data processing accelerates a wide range of research on cosmology & galaxy evolution, including elucidating the distribution of dark matter in the universe & early detection of supernova shock breakouts (supernovae during brightening that lead to major discoveries).

We are working on improving the functionality of Pwrake, which constitutes the system, to achieve further increases in processing speed. Specifically, we are working on efficient allocation of tasks using multiple cores & improving algorithms that optimize task execution order, aiming to achieve the goal of remaining large-scale experiments (approximately 28 hours of processing with 60 nodes).

Other Use Cases

HPCI Shared Storage: The lifeline of research infrastructure! A highly reliable data sharing platform achieved through geographic distribution & redundancy