JAXA Supercomputer System Annual Report February 2023-January 2024

Report Number: R23EG3212

Subject Category: Research and Development

PDF available here

• Responsible Representative: Taro Shimizu, Director, Research and Development Directorate, Research Unit III
• Contact Information: Takanori Haga, Research and Development Directorate, Research Unit III(haga.takanori@jaxa.jp)
• Members: Junya Aono, Masaharu Abe, Yu Daimon, Yuma Fukushima, Takanori Haga, Shotaro Hamato, Hiroyuki Ito, Ryohei Kirihara, Hirofumi Kurata, Yasuhiro Kawashima, Hideyo Negishi, Takenori Nakajima, Shinji Ohno, Taro Shimizu, Kei Shimura, Ryotaro Sakai, Seiji Tsutsumi, Ryoji Takaki, Kazuma Tago, Osamu Watanabe, Himeko Yamamoto

Abstract

In order to capture the unsteady phenomenon in a real-scale liquid rocket engine, the relevant physical models and numerical methods necessary for combustion LES are developed. An analysis tool is validated for the subscale test data, and applied to the development of a real-scale engine.

Reference URL

Please refer to https://stage.tksc.jaxa.jp/jedi/en/simul/index.html .

Reasons and benefits of using JAXA Supercomputer System

Since the flow and combustion in rocket chambers are in a turbulent state and have nonstationary characteristics, LES analysis is essential. Even in this verification target, analysis calculation of about several million steps is required for grid of tens to hundreds of millions of cells, so it is impossible to achieve the target without using supercomputer.

Achievements of the Year

A combustion solver LS-FLOW-HO based on the Flux Reconstruction (FR) method, which is a high-order accurate scheme, is under development to realize high-fidelity simulations of liquid rocket engine combustors on a real scale. In order to put into practical use a large-scale combustion LES that calculates several hundred coaxial injectors, we are developing a high-speed solver and a physics model to reduce the computational time. The main focus is on the development of a solver for single-phase flow with supercritical combustion pressure to be applied to the analysis of a large first-stage engine. In this fiscal year, we conducted a combustion LES of a sub-scale combustor with 42 injectors as a stepping stone to the LES of a full-scale combustor, and improved its functions including pre-post processing.

1. Development of a high-speed solver

Since speed-up tuning for the CPU (A64FX) was almost completed, this year we improved the resolution by improving the scheme (reduction in the number of computational points for the same resolution).

To suppress numerical oscillations at the interface between combustion gas and liquid oxygen (LOX), where the density ratio is several hundred times higher, we employ a limiter that mixes polynomial equations (high-order accuracy) and cell mean values (first-order accuracy) to approximate the physical quantity distribution.

Conventional solvers use a minimum value of entropy as a constraint condition for determining mixing coefficients, but the higher the degree of the approximate polynomial, the worse the resolution of the interface becomes. As in the literature (Dzanic & Witherden, JCP 2022), we considered not only the spatial distribution of entropy but also its time variation as a constraint condition, and succeeded in increasing numerical stability without excessive limiting. This enables supercritical pressure combustion simulations with a higher-order accurate scheme.

2. Development of physical model (reaction wall model)

In order to predict the heat flux at the combustor wall, a turbulent boundary layer (TBL) wall model (Muto et al, IJHMT 2019) that takes chemical reactions into account was validated. In combustion LES for sub-scale combustors, numerical instability tends to occur at low grid resolution where the TBL develops along the parallel walls of the chamber. In particular, for higher-order schemes, numerical stability was found to be reduced due to errors in the wall-normal velocity and temperature gradients on the wall modeled LES (WMLES) grid, which does not resolve the inner layer of TBL (Fukushima & Haga IJNMF 2024). We improved the evaluation method for the wall-normal gradient and introduced subgrid-scale eddy viscosity based on the assumption of an equilibrium wall model.

3. LES of a sub-scale combustor (42 injectors)

A LES analysis of a BKD combustor (LOX/GH2) at DLR was performed as a validation example of a large-scale combustion instability analysis. An overset mesh method was used to facilitate the creation of a mesh for a complex geometry with 42 coaxial injectors. To account for the coupling of acoustic eigenvalues of the injectors and chambers with heat generation fluctuations, a non-reflective boundary condition (fixed flow rate and temperature) was set upstream of the LOX and GH2 manifold geometry, and a supersonic nozzle outflow condition was used downstream. The number of computational points is approximately 1 billion. Figure 1 shows the computed results (instantaneous temperature field). To reduce the data size in post-processing, we used the in-situ visualization software Kombyne and confirmed that it works on JSS3. The calculation time required from the initial conditions until the combustion pressure became static (approximately 2 msec in physical time) was approximately 70 hours (with 1000 CPUs, 70,000 NH), and the simulation of combustion instability is expected to require about 10 times this amount of computing resources.

Fig.1: LES result for DLR BKD combustor (LOX/GH2). Instantaneous temperature and velocity distributions. About 1 billion computational points.

Publications

– Peer-reviewed papers

1) Fukushima, Y. and Haga, T., “On robust boundary treatments for wall-modeled LES with high-order discontinuous finite element methods,” International Journal for Numerical Methods in Fluids, 2024.

– Non peer-reviewed papers

1) Haga, T., Shimizu, T., Nunome, Y., “LES of Combustion Instability in Coaxial Injectors of Liquid Rocket Engines during Hydrogen Injection Temperature Ramping,” JSASS 54th Annual Meeting. (in Japanese)

2) Haga, T. and Tago, K., “GPU Acceleration of the Compressible Combustion Solver LS-FLOW-HO,” 55th FDC and 41st ANSS. (in Japanese)

3) Haga, T., “Large-Eddy Simulation of Liquid Rocket Engine Combustors Using a High-Order Unstructured Grid Method,” Keisan Kougaku, Vol. 28, No. 3. (in Japanese)

4) Haga. T., “Shock-turbulence interaction simulation by a high-order unstructured grid method satisfying positivity preserving and entropy condition,” Shockwave Symposium 2023. (in Japanese)

5) Blanchard, S. and Haga, T., “Development of a diffuse interface methodology in a high-order flux-reconstruction framework,” The 11th International Conference on Multiphase Flow.

– Oral Presentations

1) Haga, T., “GPU acceleration of a high-order combustion solver LS-FLOW-HO,” 37th CFD Symposium. (in Japanese)

2) Haga, T., “Discontinuous Capturing Schemes for LOX/GH2 Liquid Rocket Engine Combustor Simulations by Flux-Reconstruction Method,” International Conference on Spectral and High Order Methods (ICOSAHOM) 2024

Usage of JSS

Computational Information

• Process Parallelization Methods: MPI
• Thread Parallelization Methods: OpenMP
• Number of Processes: 1 – 8000
• Elapsed Time per Case: 336 Hour(s)

JSS3 Resources Used

Fraction of Usage in Total Resources*1(%): 1.98

Details

Please refer to System Configuration of JSS3 for the system configuration and major specifications of JSS3.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

ISV Software Licenses Used

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

JAXA Supercomputer System Annual Report February 2023-January 2024