JAXA Supercomputer System Annual Report February 2024-January 2025

Report Number: R24EG3212

Subject Category: Research and Development

PDF available here

• Responsible Representative: Taro Shimizu, Director, Research and Development Directorate, Research Unit III
• Contact Information: Takanori Haga, Research and Development Directorate, Research Unit III(haga.takanori@jaxa.jp)
• Members: Patrick Strempfl, Junya Aono, Masaharu Abe, Masaki Ando, Yu Daimon, Yuma Fukushima, Takanori Haga, Morimasa Hattori, Shotaro Hamato, Hiroyuki Ito, Ryohei Kirihara, Hirofumi Kurata, Hideyo Negishi, Takenori Nakajima, Shinji Ohno, Yasuhiro Ohta, Yasuhito Okano, Taro Shimizu, Kei Shimura, Seiji Tsutsumi, Ryoji Takaki, Kazuma Tago, Osamu Watanabe, Himeko Yamamoto

Abstract

In order to capture the unsteady phenomenon in a real-scale liquid rocket engine, the relevant physical models and numerical methods necessary for combustion LES are developed. An analysis tool is validated for the subscale test data, and applied to the development of a real-scale engine.

Reference URL

Please refer to 'Numerical Simulation Technology | JEDI Center(JAXA's Engineering Digital Innovation Center)'.

Reasons and benefits of using JAXA Supercomputer System

Since the flow and combustion in rocket chambers are in a turbulent state and have nonstationary characteristics, LES analysis is essential. Even in this verification target, analysis calculation of about several million steps is required for grid of tens to- hundreds of millions of cells, so it is impossible to achieve the target without using supercomputer.

Achievements of the Year

A reacting flow solver LS-FLOW-HO based on the high-order flux reconstruction (FR) method is under development to realize high-fidelity simulations of liquid rocket engine combustors on a real scale. The results of this year are as follows.

1. Tuning for vector architectures

LS-FLOW-HO is being ported and tuned for accelerators (GPUs, vector engines, etc.) that are expected to be used in future supercomputers. This year, vectorization tuning of the aerodynamic version of LS-FLOW-HO (without combustion model, real fluid model, etc.) was performed for NEC VE Type 10C.

Although it is possible to vectorize the code to some extent by specifying compiler options and simple compiler instruction lines, the vectorization rate was insufficient as it was, and performance could not be extracted. Therefore, the loop structure was reviewed, and performance was improved by increasing the vectorization rate by increasing the average vector length and by making memory accesses more efficient (Figure 1).

2. Introduction of an efficient time integration scheme (PERK)

Based on the Explicit Runge-Kutta (ERK) scheme, which can be easily parallelized, the Paired Explicit Runge-Kutta (P-ERK) scheme has been implemented in LS-FLOW-HO to relax the Courant number condition and increase the time step. P-ERK requires a region partitioning that takes into account the load balance because the computational load is spatially non-uniform. In this study, the multi-constraint function of the graph partitioning tool METIS was used.

In the benchmark case of the SD7003 airfoil model, P-ERK resulted in 4.4 times higher CFLs and 3 times faster than 3-stage ERK.

When applied to a LOX/GH2 single-element combustor as a real-world supercritical combustion problem, the CFL number and speedup by P-ERK were 3.1 and 0.79 times higher, respectively. The reason is that the computational grid used in this study has a large fraction of small cells that are rate-limiting, increasing the total computational cost. Optimization of the P-ERK parameters, i.e., cell leveling and number of stages, is expected to reduce the grid dependence.

3. Introduction of inflow disturbance by synthetic eddy method (SEM)

In the past, unsteady fluid analysis of rocket engines was performed for each element such as combustors, turbo pumps, turbines, etc. However, inflow disturbances in each computational domain were not adequately modeled. In order to accurately reproduce the flow distribution bias and manifold fluctuating pressure, the synthetic eddy method (SEM) was adopted, which is easily applicable to complex geometries. A simulation of the upstream region (RANS or LES) was performed beforehand to obtain the average velocity and Reynolds stress distribution at the inter domain boundary. This is used as the input condition for the SEM, and a velocity disturbance is applied to the inflow boundary of the downstream region, but it is common practice to provide a run-up interval before the flow develops into turbulence. A LES of turbulent parallel-plate channel was performed to determine the required grid resolution according to the approximate order of the FR scheme. The required run-up interval for the P3 scheme (4th order accuracy) is about 15 times the channel half-width. (Figure 2)

4. LES of a sub-scale combustor (42 injectors)

As a validation example of combustion instability analysis, LES of a DLR BKD combustor (LOX/GH2) with 42 coaxial-type injectors was performed. An overset grid method was used and the number of computational points was approximately 1 billion. The computation time required for the analysis with a physics time of 10 msec was approximately 650 hours (500 CPUs used, 328,000 NH).

The results (instantaneous fields of density and pressure and PSD of combustion pressure) are shown in Figure 3. 1T mode and other major chamber modes were successfully captured.

Fig.1: Performance ratio (normalized by FX1000) and computing efficiency (per peak) of LS-FLOW-HO.

Fig.2: Mean velocity and Reynold's stress profile of turbulent channel flow (〖𝑅𝑒〗_𝜏~180) using synthetic-eddy-method (SEM). x/d is the distance from the inlet boundary normalized by the channel half-width d.

Fig.3: Instantaneous fields of density and pressure and PSD of combustion pressure for DLR BKD combustor (LOX/GH2) with 42 coaxial-type injectors.

Publications

- Non peer-reviewed papers

1) Haga, T., "Acceleration of a high-order combustion solver LS-FLOW-HO using paired explicit Runge-Kutta schemes," 56th FDC and 42nd ANSS. (in Japanese)

2) Watanabe, O., Haga, T., Takaki, R., "Performance analysis of the high-order combustion solver LS-FLOW-HO in a multi-computation architecture and its performance acceleration using a vector architecture," 56th FDC and 42nd ANSS. (in Japanese)

3) Sakai, R., Haga, T., Tsutsumi, S., "Investigation of Synthetic Eddy Method as Inflow Condition in LES by a Flux Reconstruction Method," 56th FDC and 42nd ANSS. (in Japanese)

4) Haga, T., "Robust and Efficient Numerical Schemes for LES of Liquid Rocket Engine Combustor," ICCFD12, Kobe, Japan, July 14-19, 2024.

- Invited Presentations

1) Haga, T., "Liquid Rocket Engine Combustor Simulations by Flamelet-Based Model and Flux-Reconstruction Method," Emerging Trends in Computational Fluid Dynamics: Towards Industrial Applications,Jameson-Kim-Wang Symposium, 2024.

- Oral Presentations

1) Haga, T., "Speed-up of compressible combustion LES using the paired explicit Runge-Kutta schemes," 38th CFD symposium. (in Japanese)

2) Haga, T., Shimizu, T., "Large-Eddy Simulations of Subscale LOX/GH2 Rocket Combustors with Different Fuel Injection Temperatures," ICNC2024, Kyoto, Japan, May 7-10, 2024.

Usage of JSS

Computational Information

• Process Parallelization Methods: MPI
• Thread Parallelization Methods: OpenMP
• Number of Processes: 1 - 11520
• Elapsed Time per Case: 240 Hour(s)

JSS3 Resources Used

Fraction of Usage in Total Resources*1(%): 3.19

Details

Please refer to System Configuration of JSS3 for the system configuration and major specifications of JSS3.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

ISV Software Licenses Used

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

JAXA Supercomputer System Annual Report February 2024-January 2025