JAXA Supercomputer System Annual Report February 2021-January 2022

Report Number: R21EFHC0303

Subject Category: Common Business

PDF available here

• Responsible Representative: Naoyuki Fujita, Manager, JAXA, Security and Information Systems Department, Supercomputer Division
• Contact Information: Takanori Haga, Research and Development Directorate, Research Unit III(haga.takanori@jaxa.jp)
• Members: Takanori Haga, Hiroyuki Ito, Kiyoshi Kumahata, Seiji Tsutsumi

Abstract

In this work, we focus on the spatial distribution and time fluctuation of the chamber pressure and the wall heat flux assumed in the full-scale engine, and perform a complete three-dimensional large-eddy simulation (LES). The purpose of this work is to acquire knowledge that cannot be obtained by conventional simplified analysis (single or several injectors, cake-cut shape assuming symmetry in the circumferential direction). The analysis target is a full-scale combustor (more than 500 injectors) of the LE-X engine, and a large-scale combustion LES is performed using an efficient tabulated chemistry (flamelet) model. Such a LES of 100 or more injectors has never been reported, and it will be possible for the first time by combining the computing performance of the JAXA's new supercomputer JSS3 with the next-generation large-scale and high-speed solver LS-FLOW-HO developed in JAXA.

Reference URL

N/A

Reasons and benefits of using JAXA Supercomputer System

To perform LES for full-scale combustors (500 or more injectors), it is necessary to use at least billions to tens of billions of computational points and to integrate in time on the order of one million steps. Therefore, the use of computational resources which is the majority of JSS3 is essential.

Achievements of the Year

1. Objective

In the development of liquid rocket engine combustors, oscillating combustion and erosion of the combustor's inner walls due to local heat loads are risks that require attention. These events often appear for the first time in full-scale tests and are difficult to predict in advance using computational fluid dynamics (CFD) because they are unsteady and localized phenomena. In JAXA, a compressible and combustion LES (Large Eddy Simulation) solver, LS-FLOW-HO, has been developed to realize the prediction of these phenomena.

The full-scale combustor LES analysis of the LE-X engine conducted in the previous year could not achieve the desired results due to numerical instability, and the run was suspended. This year, we had the opportunity to try again using the remaining computing resources (approximately 350,000 node hours). A new method (IRP limiter) was developed to improve numerical stability.

The objective of this project is to confirm the computational stability of the physical time required to evaluate combustor pressure fluctuations and wall heat flux. Furthermore, we aim to reproduce the detailed flow field inside the combustor, which cannot be obtained from tests or conventional steady-state analysis, and to analyze large-scale spatio-temporal data to gain new insights.

2. Numerical methods

The governing equations are discretized using the flux reconstruction (FR) method, which is a high-order accurate unstructured grid method. A table-referenced Flamelet Progress Variable (FPV) model is used for the combustion model. To account for the properties of liquid oxygen (LOX) at cryogenic temperatures, the SRK equation of state for real gas and the Chung model are employed to evaluate transport coefficients.

The greatest numerical challenge is the numerical instability at the LOX-burnt gas interface, where the density ratio is several hundred times higher. To suppress unphysical numerical oscillations, a limiter that maintains positive pressure and density was used last year, but it did not provide sufficient stability. This year, based on the principle of minimum entropy, the IRP (Invariant Region Preserving) limiter, which also imposes a limit on the local entropy of the flow field, was extended to supercritical pressure combustion. Applied to a single injector LOX/GH2 coaxial jet flame, which is a benchmark problem, we confirmed that numerical instability at the LOX-burning gas interface can be suppressed and that the interface resolution is greatly improved by eliminating the need for a low-pass filter, which was previously added for stabilization.

3. Solver Speed-up

Continuing from last year, we tuned LS-FLOW-HO on TOKI-SORA (FX 1000) to be faster, and worked to reduce the computational cost of the SRK equation of state and transport coefficients, which are high-cost routines. Specifically, the terms in the SRK equation that depend on temperature and mass fractions of chemical species were incorporated into Flamelet tables, so that the tables can be referenced with a small number of parameters. Similarly, the transport coefficients are also table-referenced, reducing the need for expensive chemical mixing rule calculations.

For full-scale combustor analysis with more than 500 injectors, an overset grid method was employed in which a grid was created for each injector and superimposed on the combustor background grid. Since interpolation at the overset boundary was found to deteriorate the load balance of a large-scale parallel computation, the interpolation in each 3D coordinate direction was combined into a single matrix product to speed up the computation and improve the load balance. These speedups resulted in a 2.6-fold speedup over the previous year.

4. Full-scale combustor LES

The analysis conditions were the test case of an LE-X engine with a combustion pressure of 8.2 [MPa] and a mixing ratio of 6.4. The total number of calculation points is about 2.6 billion. 960 nodes of JSS3 TOKI-SORA were used, and 12 processes were set up in each node (4 threads/process), totaling 11520 processes. The total number of steps is approximately 640,000. The time integration is explicit, so the time step is 6.0 e-9 [s/step], which gives an analyzable physical time of about 3.8 [msec].

The results of the injector stand-alone calculation are mapped as the initial solution, and the instantaneous field after about 0.34 [msec] is shown in Figure 1. Although the flow field is not yet fully developed, a complex flame structure can be seen. The same restart solution is mapped in each row of concentric circles, so an axisymmetric structure can be seen. The pressure distribution on the injection surface is also nearly axisymmetric at this point. The flame and wall pressure distributions at about 2.1-3.0[msec] are shown in Figure 2 (movie). Looking at the pressure fluctuations on the injection surface, one can see some rotating counterclockwise and the other reflecting in the radial direction. As the rotating pressure wave passes through the injector, it can also be seen propagating upstream of the injector post. If such acoustic modes (combustor T and R modes and coupling between combustor modes and post modes) are maintained, high-frequency combustion oscillations may occur.

However, the amplitude of the pressure fluctuations observed in this simulation was very large, a level that could not occur in reality. When such pressure fluctuations reached the upstream of the injector, large mass flow fluctuations occurred, resulting in large flame deformation downstream of the injection surface, and thus stable combustion could not be reproduced. The reduced resolution of the flame in the second half of the movie is due to the effect of a stronger limiter for numerical stabilization. Actual combustors are equipped with resonators to suppress pressure fluctuations in the combustor, but these are not modeled in this simulation, and artificial treatments to dampen the large pressure fluctuations that occur in transient calculations from initial conditions are considered necessary.

Such thermoacoustic coupling considering combustor modes and post modes specific to the actual geometry is a phenomenon that can only be reproduced by full-scale analysis, and is expected to be a means of predicting combustion oscillations. Quantitative comparison with experimental data will be essential in the future. Furthermore, by introducing a wall-modelled LES that takes into account the reaction flow, which is being developed in parallel, we would like to use this tool to assess the risk of wall erosion and heat absorption performance degradation due to local heat flux increase or decrease.

Fig.1: Result of full-scale combustor LES (instantaneous flowfield about 3.8 [msec] from the restart solution).

Fig.2(video): Pressure fluctuations and flame deformation developed in the combustor (animation of about 2.1-3.0 [msec]).

Publications

- Oral Presentations

Takanori Haga, Kiyoshi Kumahata, Hiroyuki Ito and Seiji Tsutsumi, Toward large-eddy simulation of a full-scale liquid rocket engine combustor, 53rd Fluid Dynamics Conference / 39th Aerospace Numerical Simulation Symposium (ANSS), 2C04, 2021.

Usage of JSS

Computational Information

• Process Parallelization Methods: MPI
• Thread Parallelization Methods: OpenMP
• Number of Processes: 11520
• Elapsed Time per Case: 336 Hour(s)

JSS3 Resources Used

Fraction of Usage in Total Resources*1(%): 2.05

Details

Please refer to System Configuration of JSS3 for the system configuration and major specifications of JSS3.

*1: Fraction of Usage in Total Resources: Weighted average of three resource types (Computing, File System, and Archiver).

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

ISV Software Licenses Used

*2: Fraction of Usage：Percentage of usage relative to each resource used in one year.

JAXA Supercomputer System Annual Report February 2021-January 2022