JAXA Supercomputer System Annual Report February 2021-January 2022

Report Number: R21EG3212

Subject Category: Research and Development

PDF available here

• Responsible Representative: Taro Shimizu, Director, Research and Development Directorate, Research Unit III
• Contact Information: Takanori Haga, Research and Development Directorate, Research Unit III(haga.takanori@jaxa.jp)
• Members: Junya Aono, Masaharu Abe, Yu Daimon, Taroh Fukuda, Yuma Fukushima, Takanori Haga, Shotaro Hamato, Hiroyuki Ito, Kaname Kawatsu, Kiyoshi Kumahata, Hideyo Negishi, Takenori Nakajima, Shinji Ohno, Taro Shimizu, Andrea Zambon, Seiji Tsutsumi, Ryoji Takaki, Kazuma Tago, Masashi Toyama

Abstract

In order to capture the unsteady phenomenon in a real-scale liquid rocket engine, the relevant physical models and numerical methods necessary for combustion LES are developed. An analysis tool is validated for the subscale test data, and applied to the development of a real-scale engine.

Reference URL

Please refer to https://stage.tksc.jaxa.jp/jedi/en/simul/index.html .

Reasons and benefits of using JAXA Supercomputer System

Since the flow and combustion in rocket chambers are in a turbulent state and have nonstationary characteristics, LES analysis is essential. Even in this verification target, analysis calculation of about several million steps is required for grid of tens to- hundreds of millions of cells, so it is impossible to achieve the target without using supercomputer.

Achievements of the Year

A high-order accurate combustion solver, LS-FLOW-HO, was developed for large-scale LES analysis of full-scale combustors. A high-order accurate flux reconstruction method with high computational efficiency and excellent resolution of turbulent eddies is applied to diffusion combustion under supercritical pressure. The flamelet progress variable method was used for the combustion model. The higher-order accuracy scheme was challenged by numerical instability at the high-density-ratio interface between liquid oxygen and gaseous hydrogen, but by extending the stabilizing limiter based on the minimum entropy principle (which suppresses not only density and pressure but also unphysical distribution of entropy) to the flow field where the real gas effect is significant, the stability and the resolution was successfully improved. The LES result for a single-element combustor is shown in Figure 1.

Furthermore, by tuning the code for FX1000 and improving the calculation speed, we succeeded in dramatically reducing the calculation cost to 1/20 or less compared to the conventional solver LS-FLOW (from several months per case to one week per case). High-fidelity unsteady combustion analysis improves the reproducibility of unsteady combustion phenomena (e.g., coupled acoustic and combustion, vortex emissions from injectors, etc.), enabling understanding and prediction of combustor technology issues.

To improve the prediction accuracy of wall heat flux in combustion large-eddy simulation (LES), a wall-modeled LES was constructed by introducing a chemical equilibrium wall model into the combustion solver LS-FLOW-HO. A fully developed turbulent channel flow of hydrogen and oxygen combustion gases with cooled walls was computed to investigate the prediction accuracy and computational cost. As a result, the wall heat flux was predicted with a good accuracy of about 5%, while the computational cost was reduced to 1/400 of that of a conventional LES that resolves the boundary layer without using the wall model. Furthermore, the prediction accuracy in the developing thermal boundary layer (Figure 2) was also investigated, and it was found that the parameter settings can be applied in the same way as for a fully developed turbulent channel flow. On the other hand, the prediction accuracy of the wall heat flux in the early stage of development affected the development of the downstream boundary layer.

This paragraph show the example of tuning of the LS-FLOW-HO code on FX1000. The highest cost procedure is the newton iteration that updates temperature, it had 17% of total cost measured by profiler FIPP. The procedure contains innermost short loops and it loops were SIMDized by compiler. Therefore almost operations of the procedure except for SIMDized innermost loop were not SIMDized. As a result, in CPU performance analysis report, floating operation wait is dominant when the procedure running, and SIMD instruction ratio is relatively low about 40%. To improve SIMD instruction ratio, we unrolled innermost loop so that almost operations in the procedure can be SIMDized. By this modification, SIMD instruction ratio increased from 40% to 90%, and running time decreased over 50%. Figure 3 shows how decreased running time by this method. Total running time of application was about 10% decreased by this method.

Fig.1: Turbulent combustion LES of LOX/H2 single-element combustor. (temperature distribution)

Fig.2: Computational setting and instantaneous temperature field of developing thermal boundary layer. Temperature distribution at h_wm=0.1δ_0 is shown near the wall, where δ_0 is channel half width.

Fig.3: Effects of SIMD promotion on execution time reduction.

Publications

– Oral Presentations

1) Takanori Haga, Kiyoshi Kumahata, Hiroyuki Ito and Seiji Tsutsumi, Toward large-eddy simulation of a full-scale liquid rocket engine combustor, 53rd Fluid Dynamics Conference / 39th Aerospace Numerical Simulation Symposium (ANSS), 2C04, 2021.

2) Takanori Haga, Taro Shimizu, Large-eddy simulations of LOX/GH2 shear-coaxial jet flame at supercritical pressure by flux-reconstruction method, The 35th Computational Fluid Dynamics Symposium, D06-4, 2021.

3) Yuma Fukushima, Takanori Haga, Assessment of wall-modeled LES with a flux-reconstruction method for high Reynolds number turbulent flows, AIAA AVIATION 2021 Forum, AIAA Paper 2021-2750, 2021.

4) Yuma Fukushima, Takanori Haga, Wall-modeled LES of a reacting channel flows with heat flux from cooled wall, The 35th Computational Fluid Dynamics Symposium, D06-3, 2021.

5) Kiyoshi Kumahata, Takanori Haga, Ryoji Takaki, A study on performance improving of high-order unstructured grid solver based on flux reconstruction method with FX1000, 35th Computational Fluid Dynamics Symposium, E07-4, 2021.

Usage of JSS

Computational Information

• Process Parallelization Methods: MPI
• Thread Parallelization Methods: OpenMP
• Number of Processes: 32 – 11520
• Elapsed Time per Case: 168 Hour(s)

JSS3 Resources Used

Fraction of Usage in Total Resources*1(%): 7.09

Details

Please refer to System Configuration of JSS3 for the system configuration and major specifications of JSS3.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

ISV Software Licenses Used

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

JAXA Supercomputer System Annual Report February 2021-January 2022