1. Introduction

2. Framework and Components

2.1. Overview: Motivational Curriculum Learning with DPPO

We aim to solve the problem of delayed rewards by motivational curriculum learning distributed proximal policy optimization (MCLDPPO). Motivational curriculum learning differs from traditional course learning, which helps agents learn by specifying tasks ranging from simple to complex, but neural networks have the problem of plasticity loss. During the training process, as the neural network starts to converge, having a large number of neurons die leads to the loss of diversity in the neural network. If the course changes too much, it is difficult for the agent to cross over from this course to another, losing plasticity. To be able to avoid the problem of plasticity loss, we no longer design individual courses, but by add rewards that contain courses. By continually adjusting the curriculum rewards, we are able to guide agents to achieve the effects of traditional course learning. At the same time, by adding multiple curriculum rewards, we can avoid the problem of agents not being able to continue learning other curriculums after learning one curriculum. We empower agents by adding reasonable curriculum reward step by step to enable them to learn multiple courses at the same time.

Moreover, we propose a novel interruption mechanism, which interrupts the current action to obtain the current observation to re-make a maneuver decision when the agent has a significant indicator change in the observation space. With the addition of the interrupt mechanism, the agent's decision frequency is substantially improved. Figure 2 illustrates the architecture of MCLDPPO.

For a single UAV combat system, the UAV combat process is modeled as a POMDP problem, which mainly consists of a five-tuple $<S,A,R,P,\mathsf{\gamma }>$. In air combat, the state $s_t$ is expressed as the current time $t$ of being able to represent the current state of the UAV. For example, the current position of the UAV, information about fired missiles and enemy alerts. The drone uses the current observation $s_t$ to select the action $a_t$. By executing the action $a_t$ combined with the set reward function, the reward ${\mathrm{r}}_{\mathrm{t}}$ corresponding to the current action is obtained. After executing action $a_t$, the drone reaches a new state $s_{t+1}$, makes a new decision $a_{t+1}$ according to the new state, and gets the reward $r_{t+1}$ for the next state $s_t$. where $S$ consists of a series of discrete states $s_0,s_1,\dots ,s_t,s_{t+1},\dots$; A is a series of discrete actions $a_0,a_1,\dots ,a_t,a_{t+1},\dots .$;$R$is a series of discrete rewards $r_0,r_1,\dots ,r_t,r_{t+1},\dots$ ; $P$ is the state transfer function, usually $P\left(s_{t+1}|s_t,a_t\right)$ is expressed as the probability distribution of the current state action pair $\left(s_t,a_t\right)$mapping to the successor states $s_{t+1}$ that can be reached; $\mathsf{\gamma }$ is the discount factor, which is used to define the importance of future rewards.

The goal of each agent is to maximize total return, defined as:

$${ G}_t=\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k}$$

(1)

To maximize the total reward obtained, we use a policy learning approach to learn an optimal policy ${\mathsf{\pi }}^{\mathrm{*}}$ to make action decisions on agents. For adequately represent the value of the current state $s_t$ and action $a_t$, the state value function $V^{\mathsf{\pi }}\left(s_t\right)$ and the action value function $Q_{\mathsf{\pi }}\left(s_t,a_t\right)$ are used to represent them. The policy-based $\mathsf{\pi }$ state value function $V^{\pi }\left(s_t\right)$ is denoted as $V^{\pi }\left(s_t\right)=E_{\pi }\left[G_t|S=s_t\right]$. The action-value function is defined as $Q_{\pi }\left(s_t,a_t\right)=E_{\pi }\left[G_t|S=s_t,A=a_t\right]$. The Advantage function is defined as $A_{\mathsf{\pi }}\left(s,a\right)=Q_{\mathsf{\pi }}\left(s,a\right)-V_{\mathsf{\pi }}\left(s\right)$. By parameterizing the agent's policy ${\mathsf{\pi }}_{\theta }$ , an objective function can be designed to measure the superiority of the policy. $\theta$ is the parameter of the policy network.

$$J\left(\theta \right)=E_{s_0}\left[V_{{\pi }_{\theta }}\left(s_0\right)\right]$$

(2)

The objective function is derived concerning the strategy’s parameter $\mathsf{\theta }$, and after obtaining the derivative, a gradient ascent is used to maximize the objective function, optimizing the strategy. The strategy gradient aims to find an optimal strategy$\mathrm{ }\mathrm{ }{\mathsf{\pi }}^{\mathrm{*}}$ and maximize its expected payoff in the environment.

$${\pi }^{*}=\underset{\pi }{\mathrm{a}\mathrm{r}\mathrm{g} max }{E}_{\pi }\left\{\sum _t^H{\gamma }^t{r}_{t+1}\mid \mathrm{S}=s_0\right\}$$

(3)

The strategy gradient method mainly updates the parameters iteratively along the gradient direction. However, this algorithm cannot guarantee the size of each update step, potentially deteriorating the strategy due to being too significant, ultimately affecting the training. Therefore Schulman et al. 37developed a Trust region policy optimization (TRPO) algorithm that solved the random policy optimization. This algorithm finds a trust region during the update, and updating this region guarantees policy performance stability. This process relies on the Kullback-Leibler (KL) divergence between the old and new policies to measure the distance between them and overcomes the learning rate limitation, theoretically guaranteeing the monotonicity of the strategy learning performance. However, this is challenging during the TRPO computations, especially the computation of the second-order Hessian matrix, and thus Schulman et al. 38 invented the (Proximal policy optimization, PPO) algorithm using first-order derivatives.

The PPO algorithm is implemented as the PPO-Penalty and PPO-Clip, with the latter used more often. The framework used in the PPO algorithm is the Actor-Critic (AC), which ensures that the gap between the new and old parameters is not too large by directly restricting the objective function. Similarly to TRPO, the Actor-network update part of PPO maximizes the "surrogate" objective. Where Equations 4 denotes the probability ratio $r_t\left(\mathsf{\theta }\right)$ of the current strategy ${\mathsf{\pi }}_{\mathsf{\theta }}$ to the old strategy ${\mathsf{\pi }}_{{\mathsf{\theta }}_{\mathrm{k}}}$.

$$\begin{array}{c}max{ L}^{CPI}\left(\theta \right)={\widehat{E}}_t\left[r_t\left(\theta \right){\widehat{A}}_t\right]\\ r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{{\theta }_{\mathrm{k} }}\left(a_t\mid s_t\right)}\end{array}$$

(4)

The generalized advantage estimation (GAE) is used to estimate the action’s advantage value ${\widehat{A}}_t\mathrm{ }$in the state based on the temporal-Difference ($\mathrm{T}\mathrm{D}(\mathsf{\lambda }$)) concept, which replaces the GAE’s value function with the advantage function to obtain different approximate estimates by adjusting the size of $\mathsf{\lambda }$ in the advantage function.

$$L^{clip}\left(\theta \right)={\widehat{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(5)

where Equation 6 presents the clip implementation, aiming to limit the policy variation to $\left[x_{min},x_{max}\right]$ that it is not too large, and to significantly simplify the algorithm using KL loss.

$$\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left(x,x_{min},x_{max}\right)=max\left(min\left(x,x_{max}\right),x_{min}\right)$$

(6)

Entropy is a measure that represents a random variable’s uncertainty. In reinforcement learning, the agent improves its exploration ability utilizing a strategy entropy added to the Actor's loss and multiplied by an Entropy coefficient of 0.01. A strategy’s entropy can be expressed as:

$$\genfrac{}{}{0pt}{}{\mathcal{H}\left(\mathsf{\pi }\left(\cdot ∣s_t\right)\right)=-{\sum }_{a_t}\mathsf{\pi }\left(a_t∣s_t\right)\log \left(\mathsf{\pi }\left(a_t∣s_t\right)\right)}{=E_{a_t\sim \mathsf{\pi }}\left[-\log \left(\mathsf{\pi }\left(a_t∣s_t\right)\right)\right]}$$

(7)

The critic network uses the $\mathrm{T}\mathrm{D}-\mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}$ to update the network parameters, where the state’s value function $V_{\varphi }\left(s_t\right)$ is estimated. $\varphi$ is the parameter of the value network.

$$TD-Error=\gamma V_{\varphi }\left({s_t}_{+1}\right)+r_{t+1}-V_{\varphi }\left(s_t\right)$$

(8)

To speed up the agent's convergence, we use a distributed proximal policy optimization technique to train the agent, as presented in the algorithmic pseudo-code.

2.5. Motivational Curriculum Learning Reward Design

An agent’s ultimate goal is to win the game. Therefore, designing the reward function is vital during the agent’s training. Traditional reward functions usually reward the winner of the final game. However, such a strategy has very sparse rewards, making it difficult for the algorithm to converge. To overcome this problem, we define the agent's actions during an air battle that lead to the final positive or negative rewards, such as gaining the enemy's field of view and the dominance value between the two aircraft postures. The rewards of the corresponding posture are generated by evaluating the posture in the air battle.

When adding nodal event rewards, a scene has specific rewards that sufficiently encourage the agent. The rewards are divided into nodal event rewards and continuous change rewards.

Nodal event bonus: Close-range dodging enemy missiles gives a bonus. At the same time, the missiles skimming an enemy aircraft at close range force them to maneuver massively to pave the way for subsequent attacks, which are strategically significant and provide a particular reward. Adding nodal event rewards encourages the agent to explore.

Continuous change bonus: When the aircraft maneuvers, many continuous quantities change, such as speed, attitude, and velocity affecting the Speed and sideslip angle limit penalties, relative attitude advantage between aircraft, and missile threat rewards. The continuous bonus is calculated as follows:

(11)

(12)

In air combat, the speed of the UAV is particularly crucial, and $R_{velocity}$ is used to encourage the agent to maintain a high speed. To keep the UAV in a relatively well attitude, a certain penalty is given to the aircraft's sideslip angle, and $R_{\mathsf{\beta }}$ is employed to limit the aircraft's sideslip angle.

(13)

(14)

To evaluate the attitude advantage $R_{Advantage}$ between UAVs reasonably, altogether the attitude advantage bonus is evaluated from the combination of angle and distance between UAVs. Angle is the angle between two UAVs, Distance is the distance between two UAVs. $R_{Threat}$ is used to evaluate the threat of enemy missiles to UAVs, ang is the angle between missiles and UAVs, t is the missile approach time of the missile approaching the UAV. For a more reasonable evaluation of $R_{Threat}$, both $ang$and$t$take into account a certain amount of future prediction.

Figure 8. The missile threat.

Figure 8. The missile threat.

Figure 9. The Aircraft Position Advantage.

Figure 9. The Aircraft Position Advantage.

Table 2 shows the reward function designed to fully use the knowledge of air combat, but the actual performance of the agent was not satisfactory. The initial training was done in a traditional course learning manner, first by using a targeting machine to train attack skills and subsequently by using a state machine to train evasion skills. Although this training approach is effective for deep learning, it is less applicable for reinforcement learning. Because of the problem of plasticity loss of neural networks, we propose motivated course learning based on the idea of course learning. By observing the actual confrontation, we analyze its problems and guide it to do better by designing motivational rewards. And the calculation of missile threat and the size of each reward were redesigned. The specific experimental implementation process of motivational curriculum learning is shown in Section III.C. The curriculum reward function is shown in Table 3.

2.6. Design of Training System

We constructed a framework for a distributed algorithm, fully exploited its distributed capabilities, and conducted a large number of air combat simulation scenarios in parallel to generate sufficient training data.

The Unity-based simulated air combat environment runs at a default rate of 50 frames per second because it is a highly realistic real-time scenario. The air combat agent runs every ten frames, called a time step, during which the agent acquires a series of observations from the environment. The observations involve all the information available under the current agent's field of view, which is non-perfect information. Then, the agent exploits these observations to make decisions and outputs the actions the current state takes, such as attacking, pursuing, and avoiding a target.

During training, the agent is randomly initialized to any location within the synthetic environment. It should be noted that it is crucial to have a sufficiently diverse training game to ensure that the agent can be fully explored and thus ensure the robustness of the agent's strategy. Additionally, we employ the Deep LSTM core network to handle the temporal data and solve the partially observable problem in the air warfare problem.

This work aims to develop an optimal strategy for air combat, defined as a probability distribution function from the observation to the action space. Thus, we designed a complex network structure to accomplish this goal, with the neural network comprising a Deep 2048-cell LSTM network, we propose an Actor-Critic network architecture affording global and partial observation separability.

Figure 10. Architecture of the network model.

Figure 10. Architecture of the network model.

Given a policy, the agent interacts with the environment game by continuously using the current observation as input and sampling the actions from the output distribution at each time step. The complex multidimensional observations are input to the LSTM network by encoding. The temporal features are extracted using the LSTM network state, followed by predicting the currently executed policy (action and value functions) using a fully connected layer.

To guide the agent to find the optimal strategy quickly, we decompose the core features of air combat concerning the pilot’s actual combat experience and design reward functions that contain practical meanings. The latter are used as reward functions for the agent to fire missiles against enemy agents and to achieve posture dominance. During the reward function design process, based on the idea of motivational curriculum learning, several effective and practical reward functions are designed to fully alleviate the problem of sparse rewards. Designing such additional reward functions is vital for the agent’s successful training.

Specifically, we employ a policy learning approach in reinforcement learning to train the air combat agent and use a distributed proximal policy optimization algorithm to update the network. Due to the delay between launching an air missile and hitting the target, this class of problems is called the credit allocation problem. To solve the latter problem, we use GAE to estimate the dominance of these actions between subsequent actions. Hence, training is accelerated using a standard variance stabilization technique based on the estimated dominance.

The training system comprises four main parts, as shown in Figure 11. The adversarial training environment simulates the actual air combat environment through Unity. By conducting multiple parallel battlefields in the air combat environment, scheduling each air combat environment through multi-process and multi-thread management can boost the data generation process of the air combat environment by interacting with the agent hundreds or even thousands of times to accelerate the network’s convergence.

The agent acquires state observations from the air combat simulation environment, encodes them, and then generates maneuvering decisions through forward propagation. The agent is rewarded for changing its state by interacting with the air combat environment based on the acquired maneuver decisions. The data generated by this process is packaged, including the current agent state observations, maneuver decisions, rewards, and LSTM implicit layer $<\mathrm{S},\mathrm{A},\mathrm{R},\mathrm{H}>$, into a sequence of 16 steps. The packed data are sent asynchronously to an experience replay pool built from Redis. The GPU of each optimizer samples the data in the experience replay pool to obtain the mini-batch and computes its gradient. The gradients are averaged across multiple optimizers using the MPI standard function of the NVIDIA NCCL2 protocol. The average gradient is returned to each GPU model to perform its gradient descent, ensuring that the GPU models on the distributed system are updated simultaneously.

In each small batch of data with 120 samples, each sample has a sequence of 16 actions and states combination pairs. During training, we use the Adam optimizer to update the model parameters by back-propagating the computed gradient over the samples of 16 actions in the small batch. A global statistician is also designed to determine the distribution of new versions to controllers by setting the number of version updates. Finally, distributed parallel processing enables the model to evolve dynamically and quickly, and global average gradients allow the agent to enhance performance steadily.

3. Experimentation and Evaluation

3.1. Comparison with Expert-Level Bot

To reasonably evaluate the level of reinforcement learning trained agents, the experiments are conducted using a Finite State Machine (FSM) as a baseline. The FSM is modeled by an expert rule-based air combat state machine modeled on current state-of-the-art air combat knowledge. It is capable of performing complete engagements such as target acquisition, lock, shoot, and defensive tactics.

A total of three win rates are defined: final win rate, real-time win rate, and overall win rate. The final win rate is defined as the win rate per 1,000 matchups after convergence. Real-time win rate is defined as the statistical win rate per 100 games. The overall win rate is defined as the overall win rate from the initialization of the smart body to the final convergence. The specific win rate is calculated as shown in Equation 12, where C is the number of games after convergence. The sum of the rewards is the sum of the rewards of the sampled 120 sequences, and each sequence has rewards $r$ brought by 16 actions.

(15)

As a comparison, Zhou et al. employed the predictive game tree AI against FSM and conducted 1152 confrontation experiments, where the predictive game tree algorithm achieved 700 wins, with only one tie. The predictive game tree agent achieved a 60.7% win rate against FSM. At the same time, the classic algorithms DQN, Deterministic policy gradient algorithms (DDPG) 42, Soft Actor-Critic (SAC)43 in reinforcement learning are also used to fight against FSM to calculate the final winning rate. The experimental results are shown in Table 4.

We did a large number of experiments to train the agent, using the MCLDPPO algorithm to train the reinforcement learning agent. The MCLDPPO agent converged after about 40,000 games against FSM, and after convergence the real-time win rate reached a maximum of 82%, the average win rate was about 66%, and the average tie rate was about 10%. The high average tie rate is because the state machine can fly directly out of the agent's attack range after firing all missiles. Moreover, it can be considered that when FSM loses its attack capability, the agent's average win rate exceeds 70%. Compared with the predictive game tree algorithm, our method is improved, and the reward function and action can be further refined, so there is substantial room to improve the agent.

The high-resolution view of the training environment is removed in the actual training process to reduce system rendering overhead and improve system resource utilization. Figure 13a illustrates the agent launching a missile first as a feint to force the enemy to maneuver to avoid the missile by maintaining the guidance chain. A follow-up missile is fired to hit the enemy aircraft during a mass maneuver. In Figure 13b, the red plane has a bad attitude, and playing a constant game with the blue plane gains an altitude advantage. In Figure 13c, the aircraft actively seeks the enemy's view and thus seizes the opportunity to launch the missile early and maintain the missile's guidance chain to guide it. In Figure 13d, the agent conducts the diving maneuver to make a quick turnaround and gain attitude advantage, forcing the enemy to conduct a mass maneuver by firing a missile and replacing it with another missile to hit the enemy.

3.5. Impact of Key Parameters

The real performance and training speed during the actual training are impacted by many hyperparameters. Thus, this section conducts small-scale experiments to assess how several important parameters affect the agent’s actual performance. Table 7 lists the hyperparameters applied.

Batch Size: Increasing the batch size during the training process requires adding more rollouts to the experience pool and more GPU resources to optimize the larger batches. Simultaneously, utilizing a bigger batch size can speed up the model’s convergence, and theoretically, using twice the GPU requires half the training time. In practice, the linear acceleration impact only begins at the beginning of training, and the acceleration effect reduces with the subsequent iterations, as shown in Figure 27a. However, increasing the batch size saves a significant amount of training time.

Figure 27. Effect of different hyperparameters on agent performance.

Figure 27. Effect of different hyperparameters on agent performance.

Data Quality: Data reuse is defined as the number of times data can be reused, and a usage rate of 1 indicates that only the data produced by the current model are used. Data provided by the prior model affect the model’s maximum capacity. Although the top limit is greater for the 0.5 case than the 1 case, the former requires twice the time for practical testing, which is costly, as shown in Figure 27b.

Impact of sequence length: Multiple adversarial settings are simultaneous in a distributed environment, each involving numerous parallel battlefields, generating many state-action reward pairings. In the backpropagation utilizing GAE to determine the action advantage, the advantage must be truncated at the proper place, raising the question of how many actions comprise a more sensible sequence. In this plenary experiment, the default sequence length is 16 actions. As depicted in the Figure 27c, excessively long or short sequences are not very beneficial to practical training.

4. Conclusions

References

1. Ryan, H. Air Combat Evolution (ACE) [Internet]. 2019 Jun. Available online: https://www.darpa.mil/program/air-combat-evolution (accessed on 1 May 2023).
2. Sun, Z.; Yang, S.; Piao, H. A survey of air combat artificial intelligence. Acta Aeronaut. Sin. 2021, 42, 525799. [Google Scholar]
3. Burgin, G.; Owens, A. An adaptive maneuvering logic computer program for the simulation of one-to-one air-to-air combat. Volume 2: Program Description. NASA, 1975:5-10.
4. Burgin, G. Improvements to the adaptive maneuvering logic program. NASA,1986:2-6.
5. Mcmanus, J.; Goodrich, K. Application of artificial intelligence (AI) programming techniques to tactical guidance for fighter aircraft. Guidance, Navigation and Control Conference 1989, 3525.
6. Osborne, M.; Rubinstein, A. A Course in Game Theory; MIT Press: Cambridge, MA, USA, 1994; pp. 1–2. [Google Scholar]
7. Xu, G.; Liu, Q.; Zhang, H. The application of situation function in differential game problem of the air combat. Chinese Automation Congress (CAC) IEEE 2018: 1190-1195.
8. Virtanen, K.; Karelahti, J.; Raivio, T. Modeling air combat by a moving horizon influence diagram game. J. Guid. Control. Dyn. 2006, 29, 1080–1091. [Google Scholar] [CrossRef]
9. Park, H.; Lee, B.; Tahk, M. Differential game based air combat maneuver generation using scoring function matrix. Int. J. Aeronaut. Space Sci. 2016, 17, 204–213. [Google Scholar] [CrossRef]
10. Alkaher, D.; Moshaiov, A. Dynamic-escape-zone to avoid energy-bleeding coasting missile. J. Guid. Control. Dyn. 2015, 38, 1908–1921. [Google Scholar] [CrossRef]
11. Shachter, R. Evaluating influence diagrams. Oper. Res. 1986, 34, 871–882. [Google Scholar] [CrossRef]
12. Koller, D.; Milch, B. Multi-agent influence diagrams for representing and solving games. Games Econ. Behav. 2003, 45, 181–221. [Google Scholar] [CrossRef]
13. Virtanen, K.; Raivio, T.; Hamalainen, R. Modeling pilot's sequential maneuvering decisions by a multistage influence diagram. J. Guid. Control. Dyn. 2004, 27, 665–677. [Google Scholar] [CrossRef]
14. Pan, Q.; Zhou, D.; Huang, J. Maneuver decision for cooperative close-range air combat based on state predicted influence diagram. IEEE International Conference on Information and Automation (ICIA). IEEE, 2017, 726-731.
15. Zheng, H.; Deng, Y.; Hu, Y. Fuzzy evidential influence diagram and its evaluation algorithm. Knowl. -Based Syst. 2017, 131, 28–45. [Google Scholar] [CrossRef]
16. Sprinkle, J.; Eklund, J.; Kim, H. Encoding aerial pursuit/evasion games with fixed wing aircraft into a nonlinear model predictive tracking controller. 2004 43rd IEEE Conference on Decision and Control (CDC)(IEEE Cat. No. 04CH37601). IEEE, 2004, 3: 2609-2614.
17. Kaneshige, J.; Krishnakumar, K. Artificial immune system approach for air combat maneuvering. Intelligent Computing: Theory and Applications V. SPIE 2007, 6560, 68–79. [Google Scholar]
18. Wang, Y.; Huang, C.; Tang, C. Research on unmanned combat aerial vehicle robust maneuvering decision under incomplete target information. Adv. Mech. Eng. 2016, 8, 1–12. [Google Scholar] [CrossRef]
19. Ji, H.; Yu, M.; Han, Q. Research on the Air Combat Countermeasure Generation Based on Improved TIMS Model. Journal of Physics: Conference Series. IOP Publishing 2018, 1069, 012039. [Google Scholar] [CrossRef]
20. Huang, C.; Dong, K.; Huang, H. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization. Journal of Systems Engineering and Electronics 2018, 29, 86–97. [Google Scholar] [CrossRef]
21. McGrew, J. Real-time maneuvering decisions for autonomous air combat. Mass. Inst. Technol. 2008, 3–10. [Google Scholar]
22. Geng, W.; Ma, D. Study on tactical decision of UAV medium-range air combat. The 26th Chinese Control and Decision Conference (2014 CCDC). IEEE, 2014: 135-139.
23. Zuo, J.; Yang, R.; Zhang, Y.; Li, Z. Intelligence decision-making in air combat maneuvering based on heuristic reinforcement learning. Acta Aeronaut. Astronaut. Sin. 2017, 38, 321168. [Google Scholar]
24. Ping, L.; Yao, M. A deep reinforcement learning based intelligent decision method for UCAV air combat. In Asian Simulation Conference; Springer: Singapore, 2017; pp. 274–286. [Google Scholar]
25. Yang, Q.; Zhu, Y.; Zhang, J. UAV air combat autonomous maneuver decision based on DDPG algorithm. In 2019 IEEE 15th International Conference on control and automation (ICCA); IEEE: Piscataway, NJ, USA, 2019; pp. 37–42. [Google Scholar]
26. Xie, L.; Ding, D.; Wei, Z.; Xi, Z.; Tang, A. Moving Time UCAV Maneuver Decision Based on the Dynamic Relational Weight Algorithm and Trajectory Prediction. Math. Probl. Eng. 2021, 25–31. [Google Scholar]
27. Pope, A.; Ide, J.; Mićović, D. Hierarchical reinforcement learning for air-to-air combat. 2021 International Conference on Unmanned Aircraft Systems (ICUAS). IEEE, 2021: 275-284.
28. Liu, T.; Tian, B.; Ai, Y.; Li, L.; Cao, D.; Wang, F. Parallel Reinforcement Learning: A Framework and Case Study. IEEE/CAA J. Autom. Sinica 2018, 5, 827–835. [Google Scholar] [CrossRef]
29. Shi, W.; Feng, Y.; Cheng, G. Research on multi-aircraft cooperative air combat method based on deep reinforcement learning. Acta Autom. Sin. 2021, 47, 1610–1623. [Google Scholar]
30. Piao, H.; Sun, Z.; Meng, G. Beyond-Visual-Range Air Combat Tactics Auto-Generation by Reinforcement Learning, 2020 International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–8.
31. Ma, Y.; Du, X.; Sun, X. Adaptive modification of turbofan engine nonlinear model based on LSTM neural networks and hybrid optimization method. Chin. J. Aeronaut. 2022, 35, 314–332. [Google Scholar] [CrossRef]
32. Schulman, J.; Wolski, F.; Dhariwal, P. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
33. Zhou, W.; Zhu, J.; Kuang, M. Multi-UAV cooperative swarm algorithm in air combat based on predictive game tree. Sci Sin Tech 2016, 46, 79–90. [Google Scholar] [CrossRef]
34. Mnih, V.; Kavukcuoglu, K.; Silver, D. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
35. Berner, C.; Brockman, G.; Chan, B. Dota 2 with large scale deep reinforcement learning. arXiv 2019, arXiv:1912.06680. [Google Scholar]
36. Vinyals, O.; Babuschkin, I.; Czarnecki, W. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 2019, 575, 350–354. [Google Scholar] [CrossRef]
37. Schulman, J.; Levine, S.; Abbeel, P. Trust region policy optimization. International conference on machine learning. PMLR, 2015: 1889-1897.
38. Schulman, J.; Wolski, F.; Dhariwal, P. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
39. Schulman, J.; Moritz, P.; Levine, S. High-dimensional continuous control using generalized advantage estimation. arXiv 2015, arXiv:1506.02438. [Google Scholar]
40. Zhou, W.; Zhu, J.; Kuang, M. An unmanned air combat system based on swarm intelligence. Sci Sin Tech 2020, 50, 363–374. [Google Scholar] [CrossRef]
41. McGrew, J.; How, J.; Williams, B. Air-combat strategy using approximate dynamic programming. J. Guid. Control. Dyn. 2010, 33, 1641–1654. [Google Scholar] [CrossRef]
42. Silver, D.; Lever, G.; Heess, N. Deterministic policy gradient algorithms. International conference on machine learning. PMLR, 2014, 387-395.
43. Haarnoja, T.; Zhou, A.; Abbeel, P. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning. PMLR, 2018: 1861-1870.
44. Bengio, Y.; Louradour, J.; Collobert, R. Curriculum learning. Proceedings of the 26th annual international conference on machine learning. 2009: 41-48.
45. Richard, S. Maintaining Plasticity in Deep Continual Learning. Conference on Lifelong Learning Agents; 2022, August 22-24; McGill University, Canada, Montreal; 2022.
46. Barto, A.; Mahadevan, S. Recent advances in hierarchical reinforcement learning. Discret. Event Dyn. Syst. 2003, 13, 41–77. [Google Scholar] [CrossRef]
47. Kapturowski, S.; Ostrovski, G.; Quan, J. Recurrent experience replay in distributed reinforcement learning. International conference on learning representations. 2018. [Google Scholar]