JAXA Supercomputer System Annual Report April 2020-March 2021

Report Number: R20ER0201

Subject Category: Space Technology

PDF available here

• Responsible Representative: Riko Oki, Associated Principal Researcher, Space technology Directorate I, Earth Observation Research Center
• Contact Information: Space technology Directorate I, Earth Observation Research Center, Takuji Kubota(kubota.takuji@jaxa.jp)
• Members: Koji Terasaki, Shunji Kotsuki, Kaya Kanemaru, Yingwen Chen, Takuji Kubota

Abstract

This study explores an effective use of satellite data including GSMaP and GPM/DPR through an advanced ensemble data assimilation method for improving numerical weather prediction (NWP) and pioneering a new precipitation product based on an NWP model and satellite observations, named as NICAM-LETKF JAXA Research Analysis (NEXRA).

Reference URL

Please refer to 'JAXA Realtime weather watch'.

Reasons and benefits of using JAXA Supercomputer System

In this study, the JSS2 is used for the NICAM-LETKF experiments to assimilate satellite observations and to conduct NWP model forecasts. The JSS2 is a necessary infrastructure for our study to conduct massive computations for the ensemble-based data assimilation and ensemble atmospheric simulations.

Achievements of the Year

(1) GPM/DPR direct assimilation experiment

A method that can directly assimilate DPR reflectivity into numerical weather forecasting models was developed using the NICAM-LETKF. Based on this development, we continue to investigate the parameter estimation for cloud microphysical processes. In this fiscal year, model parameters related to the terminal velocity of snow in NICAM were optimized by assimilating GPM/DPR observations. A comparison between the observed DPR reflectivity and the simulated DPR reflectivity by NICAM based on the contoured frequency by temperature diagram (CFAD) indicates pronounced bias in the simulated DPR reflectivity (Figure 1), possibly due to errors associated with the microphysics scheme. Sensitivity experiments of the terminal velocity coefficients of rain, snow, and graupel to the simulated DPR reflectivity showed that there is sensitivity to the terminal velocity coeffiecient of snow. Therefore, we estimate snow terminal velocity coefficient via data assimilation and keep the default values of terminal velocity coefficients for rain and graupel as the control experiment (CTRL). The result shows a clear improvement of the simulated Ku-band and Ka-band reflectivity from the TEST experiment over those from the CTRL experiment (Figure 2).

(2) Development of NICAM-LETKF=GSMaP_RNC Seamless Forecasting System

The merge precipitation forecast product described in Kotsuki et al. (2019) has been running normally in operation mode since August 2020. We have been exploring the usefulness of implementing machine learning based method for improving precipitation nowcasting with GSMaP_RNC. We have implemented the convolutional LSTM (ConvLSTM) which is one of the deep learning methods to the GSMaP_RNC as a first trial and made modifications to the network design to further improve the performance of the nowcasting. Our preliminary result suggests that the updated version of the ConvLSTM nowcasting system is an improvement over the first trial version, although the dissipating trend of the precipitation features is still noticeable. We plan to further investigate the design of the ConvLSTM network to tackle this problem for improving the performance of the nowcasting system.

(3) Dense satellite data tend to have horizontally correlated errors. Therefore, we have developed the data assimilation system to include the horizontal observation error correlation and showed the improvements both in the analysis and forecast. In the ensemble-based Kalman filter, it is important to optimize the inflation parameter when the experimental settings such as observation distributions and ensemble size changes. In this fiscal year, we explored the sensitivity to the inflation parameters because the treatment of the observation error covariance matrix changes when we include the horizontal observation error correlations. The results show that including the horizontal observation error correlations requires larger inflation compared to the case with neglecting the observation error correlations. As a result of optimizing the inflation parameters for each experimental setup, we found that the analysis is greatly improved by including the horizontal observation error correlations. The result indicates the capability of assimilating dense satellite observations by including them in data assimilation.

(4) Porting the new NEXRA system to JSS3

The new NEXRA system which assimilates data obtained from PREPBUFR, AMSU-A, GSMaP, ATMS, and MHS has been ported to JSS3. The automatic process from preprocessing of observation data, calculation of NICAM-LETKF data assimilation cycle, to 5-day forecast is working on the JSS3. The operation time for 1 operation cycle on JSS3 is about 40 minutes. Also, an improvement of the monthly precipitation distribution is also confirmed (Fig. 3). Meanwhile, the new NEXRA system is currently operating on JSS3 with quasi-real time.

(5) The development of high resolution of NEXRA

A forecast experiment for the heavy rain event occurred in July 2020 has been conducted by using the new forecast system with 14-km resolution developed in the fiscal year 2020. The initial time of this experiment is 06Z 4th July 2020. Results showed that in the duration of the heavy rain event, large amount of water vapor is transported by the wind field. This water vapor transportation is important for the persistence of the heavy rain (Fig. 4).

Fig.1: Contoured frequency by altitude diagram (CFAD; %) of observed radar reflectivity (dBZ) from (a) Ku-band, (b) Ka-band, and (c) difference of radar reflectivity (i.e., Ku-Ka) and the corresponding simulated radar reflectivity using NICAM and the Joint Simulator (d-f).

Fig.2: Time series of observed and 6-h forecast radar reflectivity (dBZ) for (a) Ku-band and (b) Ka-band averaged over three vertical heights used for the parameter estimation. Black line, red line and black circle denote CTRL, TEST, and GPM DPR observation, respectively. The abscissa shows month/date in 2014.

Fig.3: Monthly mean horizontal distribution of precipitation for 2018.02 in NEW NEXRA. (a) NICAM guess precipitation, (b) NICAM analysis precipitation, (c) precipitation obtained from GSMaP, (d) the absolute value of difference between GSMaP NICAM guess precipitation, (e) he absolute value of difference between GSMaP NICAM guess precipitation, and (f) the difference between (e) and (d), where the bluish color indicates the region where the precipitation is improved by the data assimilation processes. Unit is mm/6hr.

Fig.4: The 6 hourly snap shots for the 14-km NEXRA forecast experiment for the heavy rain event occurred at 7th July, 2020. The initial time is 06Z04JUL2020 (UTC). Shadings and arrows denote the accumulated atmosphere water vapor (unit: kg/m2) and the horizontal wind at 1500 m height (unit: m/s), respectively.

Publications

- Peer-reviewed papers

1. 2020/11 Yashiro, H., Terasaki, K., Kawai, Y., Kudo, S., Miyoshi, T., Imamura, T., et al. (2020). A 1024-Member Ensemble Data Assimilation with 3.5-Km Mesh Global Weather Simulations. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Atlanta, Georgia: IEEE Press. https://dl.acm.org/doi/10.5555/3433701.3433703

- Invited Presentations

1. 2020/4/24 Takemasa Miyoshi, Weather Predictability and Data Assimilation: Perspectives Toward Prediction and Control in Agriculture, LSSE2020, online

2. 2020/7/12 Chen, Y.-W., Terasaki, K., Kotsuki, S., Kanemaru, K., Satoh, M., Miyoshi, T., and Kubota, T., 2020: The resent progress of the weather forecast system based on the NICAM-LETKF assimilation system, Poster, JpGU-AGU Joint Meeting, online

3. 2020/7/13 Kotsuki S., Pensoneault A., Okazaki A. and Miyoshi T.: Weight Structure of the Local Ensemble Transform Kalman Filter: A Case with an Intermediate AGCM. JpGU-AGU Joint Meeting 2020

4. 2020/9/11 Kotsuki, S., Miyoshi, T., Kondo, K. and Potthast, R.: A Local Particle Filter and Its Gaussian Mixture Extension: Experiments with an Intermediate AGCM. RIKEN Data Assimilation Seminar, online

5. 2020/9/11 Keiichi Kondo, Shunji Kotsuki, Takemasa Miyoshi, A local particle filter based on non-Gaussian statistics using an intermediate AGCM, DA seminar, online

6. 2020/9/24 Takemasa Miyoshi (RIKEN), Big Data, Big Computation, and Machine Learning in Numerical Weather Prediction, Workshop on Data Assimilation and Uncertainty Quantification at the exascale, online

7. 2020/10/6 Takemasa Miyoshi, Big Data, Big Computation, and Machine Learning in Numerical Weather Prediction, Virtual Event: ECMWF-ESA Workshop on Machine Learning for Earth System Observation and Prediction, online

8. 2020/11/27 Takemasa Miyoshi, Predicting Sudden Local Storms by 30-second-update NWP Using Phased Array Weather Radar, KU-ITB Biweekly Webinar Series, online

9. 2021/1/12 Takemasa Miyoshi,T. Honda, A. Amemiya, S. Otsuka, Y. Maejima, J. Taylor, H. Tomita, S. Nishizawa, K. Sueki, T. Yamaura, Y. Ishikawa, S. Satoh, T. Ushio, K. Koike, E. Hoshi, and K. Nakajima, Big Data Assimilation: Real-Time Demonstration Experiment of 30-s-Update Forecasting in Tokyo in August 2020, American Meteorological Society 101st Annual meeting, online

10. 2021/2/15 Yashiro, H., 2021: World's Largest Ensemble Weather Data Assimilation on Fugaku, The 3rd R-CCS International Symposium, online

11. 2021/2/17 Takemasa Miyoshi, Big Data, Big Computation, and Machine Learning in Numerical Weather Prediction, AI Chair OceaniX Webinars, IMT-Atlantique & RIKEN Online Joint Seminar Series (Jointly with Data Assimilation Seminar Series)2021

12. 2021/3/4 Takemasa Miyoshi, Fusing Big Data and Big Computation in Numerical Weather Prediction, Climate Research with HPC Forum, Supercomputing Asia 2021, Online(pre-recording)

- Oral Presentations

1. 2020/4/22 Takemasa Miyoshi, Enhancing Precipitation Prediction Algorithm by Data Assimilation of GPM Observations, The 20th meeting, online

2. 2020/5/7 Takemasa Miyoshi, Shunji Kotsuki, Koji Terasaki, Shigenori Otsuka, Ying-Wen Chen, Kaya Kanemaru, Masaki Satoh, Hisashi Yashiro, Hirofumi Tomita, Keiichi Kondo, Kozo Okamoto, Eugenia Kalnay, and Takuji Kubota, Enhancing Precipitation Prediction Algorithm by Data Assimilation of GPM Observations, EGU2020, online

3. 2020/7/13 Takemasa Miyoshi, Shunji Kotsuki, Keiichi Kondo, Roland Potthast, Local Particle Filter Implemented with Minor Modifications to the LETKF Code, JpGU-AGU Joint Meeting 2020, online

4. 2020/7/30 Takemasa Miyoshi, Enhancing Precipitation Prediction Algorithm by Data Assimilation of GPM Observations, The 21st meeting, online

5. 2020/11/4 Koji Terasaki, Takemasa Miyoshi, Including the horizontal observation error correlation in the assimilation of AMSU-A data, ECMWF/EUMETSAT NWP SAF Workshop on the treatment of random and systematic errors in satellite data assimilation for NWP, online

6. 2020/11/20 Koji Terasaki, Takemasa Miyoshi, Including the horizontal observation error correlation in the assimilation of AMSU-A data, Data assimilation seminar, online

7. 2020/12/1 Yashiro, H., Kawai, Y., Terasaki, K., Kudo, S., Miyoshi, T., Imamura, T., Minami, K., Nakano, M., Kodama, C., Satoh, M., and Tomita, H., 2020: Challenges of high-resolution large-ensemble data assimilation on supercomputer Fugaku, AGU Fall meeting, online

8. 2020/12/21 Takemasa Miyoshi RIKEN Enhancing Precipitation Prediction Algorithm by Data Assimilation of GPM Observations, The Joint PI Meeting of JAXA Earth Observation Missions FY2020, online

9. 2021/1/23 Takemasa Miyoshi, Innovating "Big Data Assimilation" technology for revolutionizing very-short-range severe weather prediction, CREST International Symposium on Big Data Application, online

10. 2021/2/5 James Taylor, Oversampling Reflectivity Observations from a Geostationary Precipitation Radar Satellite: Impact to Typhoon Forecasts within a Perfect Model OSSE Framework, ISDA, online

11. 2021/2/16 Takemasa Miyoshi, Fugaku's Illuminating a Path to the Future of Numerical Weather Prediction, Session11Distinguished Achievements in AI, Big Data and Simulations supporting Society5.0, The 3rd R-CCS International Symposium, Online

12. 2021/2/18 Takemasa Miyoshi, Big data assimilation and AI: Creating new development in real-time weather prediction, ERCIM-JST Joint Symposium on Big Data and Artificial Intelligence, Online

- Poster Presentations

1. 2020/7/12 Shigenori Otsuka, Yasumitsu Maejima, Pierre Tandeo, Takemasa Miyoshi, Toward an integrated NWP-DA-AI system for precipitation prediction, JpGU-AGU Joint Meeting 2020, online

2. 2020/7/12 Koji Terasaki and Takemasa Miyoshi, "Accounting for the horizontal observation error correlation of satellite radiances in data assimilation", JpGU, online

3. 2020/7/15 James Taylor, Atsushi Okazaki, Takumi Honda, Shunji Kotsuki, Moeka Yamaji, Takuji Kubota, Riki Oki, Toshio Iguchi, Takemasa Miyoshi, Evaluating the impact of precipitation radar observations from a geostationary satellite on typhoon forecasts, JpGU-AGU Joint Meeting 2020, online

4. 2020/7/16 Kotsuki S., Terasaki K., Satoh M. and Miyoshi T.: Ensemble-Based Data Assimilation of GPM/DPR Reflectivity into the Nonhydrostatic Icosahedral Atmospheric Model NICAM. JpGU-AGU Joint Meeting 2020

5. 2020/10/5 Shigenori Otsuka, Yasumitsu Maejima, Pierre Tandeo, and Takemasa Miyoshi, Toward an integrated NWP-DA-AI system for 30-second-update precipitation prediction. ECMWF-ESA Workshop on Machine Learning for Earth System Observation and Prediction, online

6. 2021/2/15 Jianyu Liang, Koji Terasaki and Takemasa Miyoshi, A purely data-driven approach to satellite simulator in numerical weather prediction, R-CCS International Symposium, online

7. 2021/2/15 Koji Terasaki, and T. Miyoshi, A 1024-Member Data Assimilation and Forecast Experiment with NICAM-LETKF Using Fugaku: A Heavy Rainfall Event in Kyushu in July 2020, The 3rd R-CCS international symposium, online

- Web

1. 2020/8/20 RIKEN and JAXA collaborate to offer real-time rainfall forecasts (https://www.riken.jp/en/news_pubs/news/2020/20200820_2/index.html)

2. 2020/11/24 Japanese Research Group Performs Largest Ever Meteorological Calculation on Supercomputer Fugaku (https://www.nies.go.jp/whatsnew/20201120/20201120.html)

- Other

1. Takemasa Miyoshi, Commendation by the Prime Minister for Disaster Prevention

Usage of JSS

Computational Information

• Process Parallelization Methods: MPI
• Thread Parallelization Methods: OpenMP
• Number of Processes: 5 - 400
• Elapsed Time per Case: 12 Hour(s)

Resources Used(JSS2)

Fraction of Usage in Total Resources*1(%): 0.31

Details

Please refer to System Configuration of JSS2 for the system configuration and major specifications of JSS2.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

Resources Used(JSS3)

Fraction of Usage in Total Resources*1(%): 3.24

Details

Please refer to System Configuration of JSS3 for the system configuration and major specifications of JSS3.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

JAXA Supercomputer System Annual Report April 2020-March 2021