1. Introduction

Figure 1. Six multifunction beams $j=1,2,\cdots ,6$, originating from an $i^{\mathrm{th}}$ MMPAR onboard a UAV. The parameters of all beams are different.

Figure 1. Six multifunction beams $j=1,2,\cdots ,6$, originating from an $i^{\mathrm{th}}$ MMPAR onboard a UAV. The parameters of all beams are different.

2. Multibeam Multifunction Phased Array Radar onboard UAVs

2.1. Steering and Scheduling of Multifunction Phased Array Beams

In our approach, we consider that there is an N UAVs, each of which carries an MMPAR. The MMPAR on multiple UAVs are networked through communication beams. Each MMPAR has a B steerable phased array beams. Each beam performs a given function. The radiation pattern of the $j^{\mathrm{th}}$ beam, for $j=1,2,\cdots ,B$ is given as ${\mathit{F}}_j\left(\theta ,\varphi \right)={\mathit{F}}_j^{\left(\mathrm{arr}\right)}(\theta ,\varphi )\times {\mathit{F}}^{\left(\mathrm{ant}\right)}(\theta ,\varphi )$, where ${\mathit{F}}_j\left(\theta ,\varphi \right)$ is the overall radiation pattern of the $j^{\mathrm{th}}$ beam, and $\theta$, and $\varphi$ represent the angles in the azimuth and elevation planes, respectively, ${\mathit{F}}_j^{\left(\mathrm{arr}\right)}$ and ${\mathit{F}}^{\left(\mathrm{ant}\right)}$ are the array factor of the $j^{\mathrm{th}}$ beam and antenna radiation pattern of individual elements of the array, respectively. Each $j^{\mathrm{th}}$ beam is steerable in the range $[-{\theta }_j^{\left(\lim \right)}:\Delta \theta :{\theta }_j^{\left(\lim \right)}]$ and $[-{\varphi }_j^{\left(\lim \right)}:\Delta \varphi :{\varphi }_j^{\left(\lim \right)}]$, where, ${\theta }_j^{\left(\lim \right)}$ and ${\varphi }_j^{\left(\lim \right)}$ are the scanning limits of the beam in the azimuth and elevation planes, respectively, and $|{\theta }_j^{\left(\lim \right)}|>|{\varphi }_j^{\left(\lim \right)}|\forall j$. The angular step between any two steering angles in the azimuth and elevation planes is represented as $\Delta \theta$ and $\Delta \varphi$, respectively.

Our approach uses six beams, as shown in Figure 1, denoted by $B=6$. These beams include two communication beams, a volume search beam, a cued search beam, a track beam, and a passive RF listener beam. Each beam has unique characteristics that depend on its function. For instance, the half-power beamwidth of the communication, volume search, and passive RF listener beams are larger compared to the cued search beam. The track beam has the highest angular resolution and gain, and is used to estimate the final parameters of the target. Although the large beamwidths of the volume search, communication, and passive RF listener beams allow for large spatial area coverage during a scan, the angular resolution is relatively small. Furthermore, to maintain simplicity, we have taken the steering limits of the beams to be the same in both the azimuth and elevation planes.

Algorithm 1 outlines the working of the MMPAR and beam scheduling. The six beams are scheduled adaptively based on the detected targets. At the beginning of the update interval $\delta t$, beam scheduling requests are received. The scheduling of beams for the duration $\delta t$ is determined based on the received requests, the current state of the beams, the target state, and the priority sequence. The details of functions performed and the scheduling of beams are as follows:

• There are two communication beams as shown in Figure 1 and Figure 2. One communication beam facilitates communication between the UAVs and is always available with the highest scheduling priority. The other communication beam forms a link between the ground station (GS) and the MMPAR on board the UAV, allowing the MMPAR to function as a communication relay. Both communication beams and the volume search beam operate simultaneously during a given $\delta t$. However, when a target is detected by the volume search beam, the second communication beam ceases operation, and communications from the second beam are transferred to other GS nodes.
• To improve the detection and tracking of UAVs, the volume search beam is designed with the largest coverage among all the beams. This enables the radar to scan a vast volume of space during the given time interval $\delta t$. However, once a target is detected, the scheduling priority of the volume search beam corresponding to that target is reduced for future update intervals. This approach ensures that the other beams with higher scheduling priority can focus on tracking the target with more accuracy and efficiency.
• The cued search beam is used to confirm the presence of a target detected by the volume search. Both the cued search and volume search beams can scan simultaneously.
• The tracking beam is directed towards the coordinates provided by the cued search beam after a time interval of $\delta t$. As the coverage area of the track beam is significantly larger than the physical tracks covered by a target during $\delta t$, the target remains within the coverage area. Due to its high angular and range resolution, the tracking beam can distinguish between multiple targets in the range and angular bins, and estimate their position, RCS, and Doppler estimates. Additionally, the tracking beam has a higher priority (after confirming target presence) than other beams. Targets are tracked adaptively, meaning that mobile/fast-moving targets are visited frequently compared to static/slow targets to update their state. Figure 3 illustrates the interaction of different types of beams with targets.
• The passive RF listener beam is not scheduled in a fixed sequence but rather randomly applied at different time intervals of $\delta t$. Its purpose is to detect active RF emissions from a malicious UAV, including intentional RF emissions for jamming and the RF link of the UAV for further analysis. Figure 3 illustrates the passive RF listener beam that is used to detect directed RF emissions from a malicious UAV.

Algorithm 1 Pseudo-code for beam scheduler and working of MMPAR.

1: procedure  RadarBeamScheduler   2: % Beam scheduling and radar operation.   3: % $i=1,2,\cdots ,N$ are the number of MMPAR, each onboard a UAV.   4: % Input to the procedure are the MMPAR beams and corresponding resources: ${\mathit{S}}_{i,j}^{\left(\mathrm{rad}\right)}\left(t\right),\forall i,j=1,2,\cdots ,6$, given in (2).   5: At the start of every update interval $\delta t$:   6: RF passive listener beam applied if $rand>0.5$, where $rand$ is a random generator in the range $\left[01\right)$.   7:     if No target is detected then   8:         Volume search and communication beams continue during $\delta t$.   9:     end if 10:     if a target is detected then 11:         Request the handover of current relay-based communications to the GS nodes. 12:         Store the coordinates of the target during the current volume search as $\left(x^{(\mathrm{tar},\mathrm{vol})}\left(t\right),y^{(\mathrm{tar},\mathrm{vol})}\left(t\right),z^{(\mathrm{tar},\mathrm{vol})}\left(t\right)\right)$. 13:         At the same $\delta t$, direct the cued search beam toward the coordinates of the target obtained during volume search. 14:         if cued search confirms the target then 15:            Store the coordinates of the target as $\left(x^{(\mathrm{tar},\mathrm{cue})}\left(t\right),y^{(\mathrm{tar},\mathrm{cue})}\left(t\right),z^{(\mathrm{tar},\mathrm{cue})}\left(t\right)\right)$. 16:            At the next update interval $t+\delta t$, direct the track beam toward the estimated position of the target obtained from the cued search. The coordinates of the target after applying the track beam are estimated as $\left(x^{(\mathrm{tar},\mathrm{trk})}(t+\delta t),y^{(\mathrm{tar},\mathrm{trk})}(t+\delta t),z^{(\mathrm{tar},\mathrm{trk})}(t+\delta t)\right)$. The RCS and Doppler estimates of the target are also obtained. 17:            At every next update interval (provided the target is present in the previous interval) the track beam will be assigned the highest priority. The state of the target ${\mathit{S}}_i^{\left(\mathrm{tar}\right)}$ is updated at every $\delta t$. 18:            Also, the targets are tracked adaptively, i.e., targets with high maneuverability are visited often compared to slow-moving targets. 19:         else 20:            Confirm targets again using volume search. 21:         end if 22:     else 23:         Continue using communication and search beams. 24:     end if 25:     During all $\delta t$, the communications between the MMPAR onboard UAVs continue. 26: return The target state, ${\mathit{S}}_i^{\left(\mathrm{tar}\right)}$. 27: end procedure

Figure 2. The UAV carrying the MMPAR moves in a straight line at a constant velocity. MMPAR emits volume search and communication beams when no target is detected. Once a target is detected, the communication link with ground nodes is handed over to other ground communication nodes.

Figure 2. The UAV carrying the MMPAR moves in a straight line at a constant velocity. MMPAR emits volume search and communication beams when no target is detected. Once a target is detected, the communication link with ground nodes is handed over to other ground communication nodes.

Figure 3. Two UAVs equipped with MMPAR are utilized to detect malicious UAVs operating as a swarm. The multifunction beams of MMPAR are employed for searching, tracking, passive listening (for active RF emissions), and classifying targets. However, one of the malicious UAVs in the swarm is causing interference with its active RF emissions. Additionally, one of the beams is utilized for communication between the radar platforms.

Figure 3. Two UAVs equipped with MMPAR are utilized to detect malicious UAVs operating as a swarm. The multifunction beams of MMPAR are employed for searching, tracking, passive listening (for active RF emissions), and classifying targets. However, one of the malicious UAVs in the swarm is causing interference with its active RF emissions. Additionally, one of the beams is utilized for communication between the radar platforms.

2.2. MMPAR Parameters

The radar parameters assigned to each type of beam are unique. The configurable radar and radar platform parameters are provided in Table 2. The range of radar parameters provided in the Table 2 can be represented as

$$\begin{array}{cc}\hfill P^{\left(\mathrm{TX}\right)}& =P^{(\mathrm{TX},\max )}:\Delta P^{\left(\mathrm{TX}\right)}:P^{(\mathrm{TX},\min )},\hfill \\ \hfill f^{\left(\mathrm{c}\right)}& =f^{(\mathrm{c},\max )}:\Delta f^{\left(\mathrm{c}\right)}:f^{\left(\mathrm{c},\min \right)},\hfill \\ \hfill BW& =B{W}^{\left(\max \right)}:\Delta BW:B{W}^{\left(\min \right)},\hfill \\ \hfill \tau & ={\tau }^{\left(\max \right)}:\Delta \tau :{\tau }^{\left(\min \right)},\hfill \\ \hfill {\theta }^{\left(\mathrm{b}\right)}& ={\theta }^{\left(\mathrm{b},\max \right)}:\Delta {\theta }^{\left(\mathrm{b}\right)}:{\theta }^{\left(\mathrm{b},\min \right)},\hfill \\ \hfill {\varphi }^{\left(\mathrm{b}\right)}& ={\varphi }^{\left(\mathrm{b},\max \right)}:\Delta {\varphi }^{\left(\mathrm{b}\right)}:{\varphi }^{\left(\mathrm{b},\min \right)},\hfill \\ \hfill PRF& =PR{F}^{\left(\max \right)}:\Delta PRF:PR{F}^{\left(\min \right)},\hfill \\ \hfill PC& =P{C}_k,k=1,2,\cdots ,N^{\left(\mathrm{PC}\right)},\hfill \\ \hfill Wav& =Wa{v}_k,k=1,2,\cdots ,N^{\left(\mathrm{Wav}\right)},\hfill \\ \hfill Pol& =Po{l}_k,k=1,2,\cdots ,N^{\left(\mathrm{Pol}\right)},\hfill \end{array}$$

(1)

where, $P^{(\mathrm{TX},\max )}$, $P^{(\mathrm{TX},\min )}$ are the maximum and minimum transmit power, respectively, and $\Delta P^{\left(\mathrm{TX}\right)}$ represents the step change in the transmit power, $f^{(\mathrm{c},\max )}$ and $f^{(\mathrm{c},\min )}$ is the maximum and minimum center frequency, respectively, and $\Delta f^{\left(\mathrm{c}\right)}$ is the step change in the center frequency. The maximum and minimum bandwidth ranges are represented by $B{W}^{\left(\max \right)}$, and $B{W}^{\left(\min \right)}$, respectively, and $\Delta BW$ is the step change in the bandwidth, ${\tau }^{\left(\max \right)}$, and ${\tau }^{\left(\min \right)}$ are the maximum and minimum pulsewidths, respectively, and $\Delta \tau$ is the step change between the pulsewidth values, the maximum and minimum azimuth half-power beamwidths are represented as ${\theta }^{\left(\mathrm{b},\max \right)}$ and ${\theta }^{\left(\mathrm{b},\min \right)}$, respectively, and $\Delta {\theta }^{\left(\mathrm{b}\right)}$ is the step change of the beamwidth in the azimuth plane, and ${\varphi }^{\left(\mathrm{b},\max \right)}$, ${\varphi }^{\left(\mathrm{b},\min \right)}$ are the maximum and minimum half-power beamwidths in the elevation plane, and $\Delta {\varphi }^{\left(\mathrm{b}\right)}$ is the step change of the beamwidth in the elevation plane. The maximum and minimum range of the PRF is represented as $PR{F}^{\left(\max \right)}$ and $PR{F}^{\left(\min \right)}$, respectively, and $\Delta PRF$ is the step change in the PRF, $N^{\left(\mathrm{PC}\right)}$ is the number of intrapulse modulation options available, $N^{\left(\mathrm{Wav}\right)}$ is the number of waveform types available, and $N^{\left(\mathrm{Pol}\right)}$ is the number of polarization options available. Moreover, the range of (the magnitude of) the velocity of an $i^{\mathrm{th}}$ airborne radar platform (UAV) is given as $v_i^{(\mathrm{rad},\max )}:v_i^{(\mathrm{rad},\min )}$.

Different types of beams are assigned different radar parameters from (1). The parameters assigned to simultaneously scheduled beams should be smaller than or equal to the total resources of the parameters available. The parameters assigned to beams remain the same unless a change is recommended by the cognitive block. Let ${\mathit{S}}_{(i,j)}^{\left(\mathrm{rad}\right)}\left(t\right)$ represent the $j^{\mathrm{th}}$ beam at the $i^{\mathrm{th}}$ radar onboard UAV at time t. The beam ${\mathit{S}}_{(i,j)}^{\left(\mathrm{rad}\right)}\left(t\right)$ and the corresponding radar parameters assigned are given as

$$\begin{array}{c}{\mathit{S}}_{(i,j)}^{\left(\mathrm{rad}\right)}\left(t\right)=[P_{(i,j)}^{\left(\mathrm{TX}\right)}\left(t\right)f_{(i,j)}^{\left(\mathrm{c}\right)}\left(t\right)B{W}_{(i,j)}\left(t\right){\tau }_{(i,j)}\left(t\right)\hfill \\ G_{(i,j)}\left({\theta }^{\left(\mathrm{b}\right)}\left(t\right),{\varphi }^{\left(\mathrm{b}\right)}\left(t\right)\right)P_{(i,j)}^{\left(\mathrm{av}\right)}\left(t\right)PR{F}_{(i,j)}\left(t\right)P{C}_{(i,j)}\left(t\right)\hfill \end{array}$$

$$\begin{array}{c}\hfill Wa{v}_{(i,j)}\left(t\right)Po{l}_{(i,j)}\left(t\right)].\end{array}$$

(2)

Table 2. Radar and radar platform parameters.

Table 2. Radar and radar platform parameters.

Serial #	Radar parameter	Representation
1	Transmit power	$P^{\left(\mathrm{TX}\right)}$
2	Center frequency	$f^{\left(\mathrm{c}\right)}$
3	Bandwidth	$BW$
4	Pulsewidth	$\tau$
5	Antenna gain based on beamwidths in the azimuth and elevation planes	$G\left({\theta }^{\left(\mathrm{b}\right)},{\varphi }^{\left(\mathrm{b}\right)}\right)$
6	Pulse repetition frequency	$PRF$
7	Power aperture product, (where $A\left({\theta }^{\left(\mathrm{b}\right)},{\varphi }^{\left(\mathrm{b}\right)}\right)$ is the subarray aperture size)	$P^{\left(\mathrm{av}\right)}=P^{(\mathrm{TX},\max )}\tau PRFA\left({\theta }^{\left(\mathrm{b}\right)},{\varphi }^{\left(\mathrm{b}\right)}\right)$
8	Intrapulse modulation/pulse compression	$PC$
9	Waveform type	$Wav$
10	Polarization	$Pol$
11	Radar platform position	($x^{\left(\mathrm{rad}\right)}$, $y^{\left(\mathrm{rad}\right)}$, $z^{\left(\mathrm{rad}\right)}$)
12	Radar platform velocity	${\overrightarrow{v}}^{\left(\mathrm{rad}\right)}$

3. Reinforcement Learning in our Approach

3.3. Multi-agent Q-Learning

If an anomaly persists for a duration of $n\delta t$, then corrective action is required to remove the anomaly. Such corrective action involves changing the runtime radar parameters and/or repositioning the radar platform. Table 4 provides a list of possible actions to remove the anomalies listed in Table 3. The combined set of available actions ${\mathit{A}}_i\left(t\right)$ at a given time t and at the $i^{\mathrm{th}}$ MMPAR can be represented as

$$\begin{array}{c}\hfill {\mathit{A}}_i\left(t\right)=\left[{\mathit{S}}_i^{\left(\mathrm{rad}\right)}\left(t\right)\left(x^{\left(\mathrm{rad}\right)},y^{\left(\mathrm{rad}\right)},z^{\left(\mathrm{rad}\right)}\right)\right].\end{array}$$

(10)

Choosing an action from Table 4 may not result in an optimal outcome. It is preferable to select an action that can maximize current and future returns, i.e., remove current and potential future anomalies with minimum overhead. After an action is taken, we can reach three states given in Table 5. Each state has an associated fixed reward. There is also an overhead for each action represented by O. The total reward is the addition of a fixed reward for the state action pair and overhead for the action shown in Table 5. This reward is used to calculate the Q-value given in Algorithm 2. The optimum Q-values are obtained after multiple iterations of training. Overall, the anomalies, corresponding optimum actions, and Q-values are listed in Table 6.

Algorithm 2 Pseudo-code for multi-agent RL algorithm using Q-learning and MMPAR data.

1: procedure  StateAnomalyActionReward   2: % The input to the algorithm are target states ${\mathit{S}}^{\left(\mathrm{tar}\right)}\left(t\right)$, radar parameters ${\mathit{S}}^{\left(\mathrm{rad}\right)}\left(t\right)$, radar platform position ($x^{\left(\mathrm{rad}\right)}$, $y^{\left(\mathrm{rad}\right)}$, $z^{\left(\mathrm{rad}\right)}$), and policy for a given scenario.   3:     Consider that we are at $i^{\mathrm{th}}$ MMPAR onboard a UAV.   4:     if ${\mathit{S}}_i^{\left(\mathrm{tar}\right)}\left(t\right)$ and ${\mathit{S}}_i^{\left(\mathrm{tar}\right)}(t-\delta t:t-n\delta t)$ do not conform to the policy for anomaly then   5:         Identify the anomaly.   6:         if Anomaly is already present in the anomaly list then   7:            For a $k^{\mathrm{th}}$ anomaly, $p^{\mathrm{th}}$ state, reward r, and $j^{\mathrm{th}}$ action from Table 5, the corresponding Q-value is calculated as:   8:            $\mathrm{New}$$Q\left({\mathit{S}}_{i,k,p}^{\left(\mathrm{tar}\right)}\left(t\right),a_{i,k,p,j}\left(t\right)\right)=Q\left({\mathit{S}}_{i,k,p}^{\left(\mathrm{tar}\right)}\left(t\right),a_{i,k,p,j}\left(t\right)\right)+\beta [r_{i,k,p,j}\left(t\right)\left({\mathit{S}}_{i,k,p}^{\left(\mathrm{tar}\right)}\left(t\right),a_{i,k,p,j}\left(t\right)\right)+\gamma \mathrm{maxQ}$${}^{\prime }\left({{\mathit{S}}^{\prime }}_{i,k,p}^{\left(\mathrm{tar}\right)}\left(t\right),a_{i,k,p,j}^{\prime }\left(t\right)\right)-Q\left({\mathit{S}}_{i,k,p}^{\left(\mathrm{tar}\right)}\left(t\right),a_{i,k,p,j}\left(t\right)\right)]$     % where $\beta$ is the learning rate, r is the reward, $\gamma$ is the future discount rate, and $Q^{\prime }$ represents maximum expected future rewards given a future state ${\mathit{S}}^{\prime }$, and action $a^{\prime }$.   9:            Update the Q-learning table. 10:         else 11:            Update the anomaly list. 12:         end if 13:     end if 14: return Q-learning table given in Table 6. 15: end procedure

Algorithm 3 Pseudo-code for evaluation and comparison of RL-based approach with other AI algorithms.

1: procedure  Evaluation-RL-AI   2: % For RL evaluation   3:     Create a dynamically changing scenario by introducing multiple targets and randomness representing noise and clutter, and active RF interference emissions.   4:     Identify anomalies and select optimum action based on Q-value. See Table 6.   5:     if Anomaly persists after an optimum action then   6:         Take the action with lesser priority (smaller Q-value than the previous).   7:         if Anomaly is removed after lesser priority action then   8:            Update the Q-learning table with an updated Q-value for the new action.   9:         else 10:            Select another action with a lower priority and repeat. 11:         end if 12:     end if 13:     Calculate the percentage of recurrence of anomalies after corrective actions are applied. 14: % For supervised AI algorithms 15:     For an anomaly, the AI algorithm is trained with an optimum action label based on the target state ${\mathit{S}}_i^{\left(\mathrm{tar}\right)}\left(t\right)$ and radar parameters ${\mathit{S}}_i^{\left(\mathrm{rad}\right)}\left(t\right)$. 16:     Calculate the percentage of recurrence of anomalies for different supervised AI algorithms. 17: return Percentage recurrence of anomalies for RL and supervised AI algorithms. 18: end procedure

Algorithm 2 provides the implementation of multi-agent Q-learning using MMPAR data. Each MMPAR onboard a UAV acts as an RL agent. The state of each agent is represented by the current and previous states of the targets after an anomaly is detected. Q-values are learned using the Q-learning algorithm and rewards from the environment for a given state-action pair. The optimum action based on Q-value is updated in Table 6. To evaluate the Q-learning table in a simulated environment scenario, multiple targets, noise, clutter, and active RF emissions as interference are introduced to create a complex and dynamic scenario. Algorithm 3 outlines the evaluation procedure.

A Markov decision process (MDP) is a mathematical framework used to model decision-making problems in situations where outcomes are partly random and partly under the control of a decision maker. In the case of the presented approach, the MDP represents the states of the targets, the occurrence of anomalies, the actions taken to remove the anomalies, and the corresponding Q-values shown in Figure 5.

Table 4. Possible actions to eliminate anomalies for accurate detection, tracking, and classification of UAVs in the swarm.

Table 4. Possible actions to eliminate anomalies for accurate detection, tracking, and classification of UAVs in the swarm.

Serial #	Action	Action label
1	Change of center frequency	$a_1$
2	Change of pulsewidth	$a_2$
3	Change of bandwith	$a_3$
4	Change of PRF	$a_4$
5	Introduce intrapulse modulation or change of type of intrapulse modulation	$a_5$
6	Change of number of pulses	$a_6$
7	Change in transmit power	$a_7$
8	Change of antenna beamwidth	$a_8$
9	Change in the polarization of the phased array	$a_9$
10	Change in type of sounding signal	$a_{10}$
11	Changes in the beam scheduling	$a_{11}$
12	Change of position of the radar platform onboard UAV	$a_{12}$

Figure 5. A Markov decision process is used to represent our approach. Initially, the state of the target is examined for anomalies at a particular radar onboard UAV. If an anomaly is detected, a set of actions is available to correct the anomaly and transition to the desired state. Each action corresponds to a Q-value. The action with the highest Q-value is selected.

Figure 5. A Markov decision process is used to represent our approach. Initially, the state of the target is examined for anomalies at a particular radar onboard UAV. If an anomaly is detected, a set of actions is available to correct the anomaly and transition to the desired state. Each action corresponds to a Q-value. The action with the highest Q-value is selected.

The goal of the MDP is to achieve an anomaly-free state for radar operation. If an anomaly arises, the best action is chosen based on the optimum Q-value to remove the anomaly and move to the desired state as shown in Figure 5. If the action based on an optimum Q-value is not able to remove the anomaly, the Q-value is updated for the state-action pair using a negative reward. A second priority action is used to remove the anomaly if the first action fails, and the process repeats. By using an MDP, the approach can systematically and optimally address anomalies in radar operation.

Table 5. There are six anomalies identified in our approach. Three states can arise after an anomaly occurs and an action is taken. Rewards are assigned for each state and action. The total reward for an action corresponding to a state is the addition of a fixed reward and overhead of the action represented by O. The Q-values are calculated using this reward (see Algorithm 2).

Table 5. There are six anomalies identified in our approach. Three states can arise after an anomaly occurs and an action is taken. Rewards are assigned for each state and action. The total reward for an action corresponding to a state is the addition of a fixed reward and overhead of the action represented by O. The Q-values are calculated using this reward (see Algorithm 2).

Anomalies	States	${\mathbf{a}}_1$	${\mathbf{a}}_2$		${\mathbf{a}}_{\mathbf{N}}^{\left(\mathbf{a}\right)}$
	Anomaly not removed	$\left[r_{1,1,1}=-10+O_1,Q_{1,1}\right]$	[$r_{1,1,2}=-10+O_2,Q_{1,2}]$	⋯	$\left[r_{1,1,N^{\left(\mathrm{a}\right)}}=-10+O_{N^{\left(\mathrm{a}\right)}},Q_{1,N^{\left(\mathrm{a}\right)}}\right]$
Anomaly(1)	Anomaly removed but reappears after $t>\delta t$	$\left[r_{1,2,1}=5+O_1,Q_{1,1}\right]$	[$r_{1,2,2}=5+O_2,Q_{1,2}]$	⋯	$\left[r_{1,2,N^{\left(\mathrm{a}\right)}}=5+O_{N^{\left(\mathrm{a}\right)}},Q_{1,N^{\left(\mathrm{a}\right)}}\right]$
	Anomaly removed and does not reappear	$\left[r_{1,3,1}=8+O_1,Q_{1,1}\right]$	[$r_{1,3,2}=8+O_2,Q_{1,2}]$	⋯	$\left[r_{1,3,N^{\left(\mathrm{a}\right)}}=8+O_{N^{\left(\mathrm{a}\right)}},Q_{1,N^{\left(\mathrm{a}\right)}}\right]$
⋮	⋮	⋮	⋮	⋯	⋮
	Anomaly not removed	$\left[r_{6,1,1}=-10+O_1,Q_{6,1}\right]$	[$r_{6,1,2}=-10+O_2,Q_{6,2}]$	⋯	$\left[r_{6,1,N^{\left(\mathrm{a}\right)}}=-10+O_{N^{\left(\mathrm{a}\right)}},Q_{6,N^{\left(\mathrm{a}\right)}}\right]$
Anomaly(6)	Anomaly removed but reappears after $t>\delta t$	$\left[r_{6,2,1}=5+O_1,Q_{6,1}\right]$	[$r_{6,2,2}=5+O_2,Q_{6,2}]$	⋯	$\left[r_{6,2,N^{\left(\mathrm{a}\right)}}=5+O_{N^{\left(\mathrm{a}\right)}},Q_{6,N^{\left(\mathrm{a}\right)}}\right]$
	Anomaly removed and does not reappear	$\left[r_{6,3,1}=8+O_1,Q_{6,1}\right]$	[$r_{6,3,2}=8+O_2,Q_{6,2}]$	⋯	$\left[r_{6,3,N^{\left(\mathrm{a}\right)}}=8+O_{N^{\left(\mathrm{a}\right)}},Q_{6,N^{\left(\mathrm{a}\right)}}\right]$

Table 6. Anomaly and possible actions to remove that anomaly in a given scenario. The rewards and Q-values corresponding to actions are also provided.

Table 6. Anomaly and possible actions to remove that anomaly in a given scenario. The rewards and Q-values corresponding to actions are also provided.

	Anomaly (observed from target returns)	Actions	Q-values
		$a_2$↓	$Q_{1,2}$
		$a_3$↑	$Q_{1,3}$
	Number of detected targets changing at $n\delta t$	$a_5$	$Q_{1,5}$
		$a_8$↓	$Q_{1,8}$
		$a_{12}$	$Q_{1,12}$
		$a_1$$\uparrow \downarrow$	$Q_{2,1}$
		$a_6$↑	$Q_{2,6}$
	Targets detected with SNR below the threshold (either due to clutter or interference) at $n\delta t$	$a_7$↑	$Q_{2,7}$
		$a_{12}$	$Q_{2,12}$
		$a_1$	$Q_{3,1}$
	Velocity variations above a threshold at $n\delta t$	$a_4$↑	$Q_{3,4}$
		$a_{12}$	$Q_{3,12}$
Scenario		$a_1$$\uparrow \downarrow$	$Q_{4,1}$
		$a_8$↓	$Q_{2,8}$
		$a_4$↑	$Q_{4,4}$
	RCS fluctuations above a threshold at $n\delta t$	$a_8$↓	$Q_{4,8}$
		$a_9$	$Q_{4,9}$
		$a_{12}$	$Q_{4,12}$
		$a_1$$\uparrow \downarrow$	$Q_{5,1}$
	Active RF emissions from a malicious UAV	$a_3$↓	$Q_{5,3}$
		$a_{12}$	$Q_{5,12}$
		$a_2$↓	$Q_{6,1}$
		$a_3$↑	$Q_{6,3}$
		$a_5$	$Q_{6,5}$
	Classification of targets changing at $n\delta t$	$a_6$↑	$Q_{6,6}$
		$a_7$↑	$Q_{6,7}$
		$a_8$↓	$Q_{6,8}$
		$a_{12}$	$Q_{6,12}$

4. Simulation Setup and Results

4.1. Simulation Setup

The simulations are carried out in Matlab. The MMPAR onboard UAV is generated using the Matlab Radar Toolbox. The parameters of the radar are provided in Table 7. The radar resources shared by different beams (obtained from the range of parameters in Table 7) are provided in Figure 8. The maximum azimuth and elevation scan limits are [$-{55}^{\circ }:{55}^{\circ }$] and [$0^{\circ }:-{35}^{\circ }$] in the azimuth and elevation planes, respectively. The update rate for the system simulations is $f^{\left(\mathrm{u}\right)}=20$ Hz, and correspondingly the update interval is $\delta t=50$ ms.

We are considering two UAV platforms, each equipped with an MMPAR. These UAVs are labeled as UAV1 and UAV2, respectively. They follow a straight-line trajectory at a constant velocity towards the targets. UAV1 has six targets in front of it, while UAV2 scans three targets. All targets are airborne and within the range of the radars. Figure 6 shows a snapshot of UAV1 carrying MMPAR scanning the six targets using volume, cued, and track beams. In Figure 7, an airborne platform carrying a malicious RF emitter emits a beam towards UAV2 carrying MMPAR.

The targets have assigned trajectories determined by their initial and final waypoints. The targets move with an average displacement of $26.6$ km, $31.7$ km, and $1.3$ km in the x, y, and z coordinates, respectively, and their average velocity is 450 m/s. The RCS values of the targets are 1 m² and 3 m², and the Swerling1 RCS fluctuation model is employed for the simulations. The active RF emitter carried by one of the targets in Figure 7 has an EIRP of 200 dBi.

Figure 6. Volume search, cued search, and track beams are utilized by MMPAR onboard UAV1 to detect and track targets, with the six targets following distinct trajectories in the scenario.

Figure 6. Volume search, cued search, and track beams are utilized by MMPAR onboard UAV1 to detect and track targets, with the six targets following distinct trajectories in the scenario.

Figure 7. Onboard a malicious UAV, there is an active RF emitter. Meanwhile, the MMPAR onboard UAV2 generates a passive RF listener beam that can detect active RF emissions and determine their direction of arrival. This includes both direct (from active RF emitter) and reflected emissions from a target within the range of the active RF beam. The directions of arrival (DoA) are represented by straight lines.

Figure 7. Onboard a malicious UAV, there is an active RF emitter. Meanwhile, the MMPAR onboard UAV2 generates a passive RF listener beam that can detect active RF emissions and determine their direction of arrival. This includes both direct (from active RF emitter) and reflected emissions from a target within the range of the active RF beam. The directions of arrival (DoA) are represented by straight lines.

Figure 8. Resource distribution among the beams. The PAP presented here is for a peak transmit power of $150\times {10}^3$.

Figure 8. Resource distribution among the beams. The PAP presented here is for a peak transmit power of $150\times {10}^3$.

4.2. Results and Analysis

The simulations introduce anomalies to the radar environment by adding randomness according to (9). The simulation duration is 200 s, comprising 4000 update intervals. The first two anomalies and their respective priority actions for the MMPAR onboard UAV1 are shown in Figure 9 and Figure 10. For the first anomaly in Figure 9, the goal is to have above 300 correctly detected targets throughout the simulation duration, and if the number of correctly detected targets is 300 or less, it is considered an anomaly. The priority actions taken to remove the anomaly include increasing the peak power, reducing the beamwidth, increasing the number of pulses, and bandwidth. The peak power varies between $150\times {10}^3$ and $400\times {10}^3$, the beamwidth is reduced by 1 degree in both the azimuth and elevation planes, the number of pulses is increased from 10 to 35, and the bandwidth increases from $10\times {10}^6$ to $35\times {10}^6$. The results show that the priority actions help to increase the number of correctly detected targets above the anomaly threshold of 300.

In Figure 10, the second anomaly from Table 6 is shown, and we aim to keep the number of targets detected above the SNR threshold of 10 dB above 300. The three priority actions selected based on the highest Q-values include increasing peak power, decreasing the beamwidth, and increasing the number of pulses. As illustrated in Figure 10, applying the priority actions increases the number of occurrences when the SNR of the detected targets is above the threshold. The other anomalies and corresponding actions from Table 6 can also be plotted.

Figure 9. Anomaly1 at MMPAR onboard UAV1 occurs when the number of correctly detected targets over a fixed simulation duration is less than 300. Corresponding actions are taken to remove the anomaly, which include varying the peak power in the range of $150\times {10}^3$ W: $50\times {10}^3:400\times {10}^3$ W, the beamwidth $\left[23\right]$ is reduced by $1^{°}$ as [$\left[22\right]$$\left[12\right]$$\left[11\right]$], the number of pulses is changed as $10:5:35$, and the range of bandwidth change is as follows: $10\times {10}^6$ Hz: $5\times {10}^6:35\times {10}^6$ Hz. These actions help increase the number of correctly detected targets.

Figure 9. Anomaly1 at MMPAR onboard UAV1 occurs when the number of correctly detected targets over a fixed simulation duration is less than 300. Corresponding actions are taken to remove the anomaly, which include varying the peak power in the range of $150\times {10}^3$ W: $50\times {10}^3:400\times {10}^3$ W, the beamwidth $\left[23\right]$ is reduced by $1^{°}$ as [$\left[22\right]$$\left[12\right]$$\left[11\right]$], the number of pulses is changed as $10:5:35$, and the range of bandwidth change is as follows: $10\times {10}^6$ Hz: $5\times {10}^6:35\times {10}^6$ Hz. These actions help increase the number of correctly detected targets.

Figure 10. Anomaly2 occurs when the number of targets detected with an SNR above the 10 dB threshold is less than 300. Similar actions to those in Figure 9 are taken to remove the anomaly and increase the number of targets detected with an SNR above 10 dB.

Figure 10. Anomaly2 occurs when the number of targets detected with an SNR above the 10 dB threshold is less than 300. Similar actions to those in Figure 9 are taken to remove the anomaly and increase the number of targets detected with an SNR above 10 dB.

The MMPAR onboard UAV2 uses a passive RF listener beam to detect active RF emissions from a target UAV, as depicted in Figure 7. The frequency sweep is performed using the passive listener beam to detect the active emissions, and the direction of arrival is determined by the same beam, shown as a line in the figure. Another line in Figure 7 shows the detection caused by the reflection of RF energy from a non-emitting target within the emitter beam footprint towards the RF listener beam.

4.3. Comparison of RL with other AI Algorithms

In addition to RL, we also utilize the four supervised AI algorithms discussed in Section 3.4 to address anomalies. These algorithms predict the best action to remove the anomaly, and we compare their effectiveness with the Q-learning RL algorithm. Specifically, we compare the percentage of anomaly recurrence after applying the optimal action, as outlined in Algorithm 3). Figure 11 shows the results of this comparison.

Figure 11 shows that RL outperforms other supervised AI algorithms in handling anomalies. This is mainly due to RL’s continuous situational awareness from environment feedback in the form of rewards and the ability to update its policy during runtime. In contrast, once a supervised AI algorithm is trained, it tries to find the best match based on training without taking any runtime feedback from the environment. Furthermore, the performance of supervised AI algorithms is expected to decrease in complex and dynamic scenarios with rapidly changing environments. Additionally, the recurrence percentage of anomalies is higher for all AI algorithms when six targets are present compared to three targets.

Figure 11. The percentage of anomaly recurrence after taking optimum actions based on Q-values is presented in the graph. The simulations were conducted over a duration of 200 seconds. It is observed that the highest percentage of anomaly recurrence occurs when no AI algorithm is used, while the lowest percentage is obtained using RL. Additionally, the percentage of anomaly recurrence is higher for six targets as compared to three targets.

Figure 11. The percentage of anomaly recurrence after taking optimum actions based on Q-values is presented in the graph. The simulations were conducted over a duration of 200 seconds. It is observed that the highest percentage of anomaly recurrence occurs when no AI algorithm is used, while the lowest percentage is obtained using RL. Additionally, the percentage of anomaly recurrence is higher for six targets as compared to three targets.

5. Conclusions and Future Work

References

1. Khawaja, W.; Guvenc, I.; Matolak, D.W.; Fiebig, U.C.; Schneckenburger, N. A Survey of Air-to-Ground Propagation Channel Modeling for Unmanned Aerial Vehicles. IEEE Commun. Surv. & Tut. 2019, 21, 2361–2391. [Google Scholar] [CrossRef]
2. Hall, O.; Wahab, I. The use of drones in the spatial social sciences. Drones 2021, 5, 112. [Google Scholar] [CrossRef]
3. Beloev, I.H. A review on current and emerging application possibilities for unmanned aerial vehicles. Acta technologica agriculturae 2016, 19, 70–76. [Google Scholar] [CrossRef]
4. Alghamdi, Y.; Munir, A.; La, H.M. Architecture, Classification, and Applications of Contemporary Unmanned Aerial Vehicles. IEEE Consum. Electron. Mag. 2021, 10, 9–20. [Google Scholar] [CrossRef]
5. Custers, B. Future of Drone use; Springer, 2016. [Google Scholar]
6. Khawaja, W.; Semkin, V.; Ratyal, N.I.; Yaqoob, Q.; Gul, J.; Guvenc, I. Threats from and countermeasures for unmanned aerial and underwater vehicles. Sensors 2022, 22, 3896. [Google Scholar] [CrossRef] [PubMed]
7. Guvenc, I.; Koohifar, F.; Singh, S.; Sichitiu, M.L.; Matolak, D. Detection, Tracking, and Interdiction for Amateur Drones. IEEE Commun. Mag. 2018, 56, 75–81. [Google Scholar] [CrossRef]
8. Svanström, F.; Alonso-Fernandez, F.; Englund, C. Drone Detection and Tracking in Real-Time by Fusion of Different Sensing Modalities. Drones 2022, 6, 317. [Google Scholar] [CrossRef]
9. Ezuma, M.; Ozdemir, O.; Anjinappa, C.K.; Gulzar, W.A.; Guvenc, I. Micro-UAV Detection with a Low-Grazing Angle Millimeter Wave Radar. In Proceedings of the Proc. IEEE Radio and Wireless Symp. (RWS); 2019; pp. 1–4. [Google Scholar] [CrossRef]
10. Gong, J.; Yan, J.; Li, D.; Kong, D. Detection of Micro-Doppler Signals of Drones Using Radar Systems with Different Radar Dwell Times. Drones 2022, 6, 262. [Google Scholar] [CrossRef]
11. Jian, M.; Lu, Z.; Chen, V.C. Drone detection and tracking based on phase-interferometric Doppler radar. In Proceedings of the Proc. IEEE Radar Conf. 2018; pp. 1146–1149. [Google Scholar] [CrossRef]
12. Hoffmann, F.; Ritchie, M.; Fioranelli, F.; Charlish, A.; Griffiths, H. Micro-Doppler based detection and tracking of UAVs with multistatic radar. In Proceedings of the Proc. IEEE Radar Conf. 2016; pp. 1–6. [Google Scholar] [CrossRef]
13. Aldowesh, A.; Shoaib, M.; Jamil, K.; Alhumaidi, S.; Alam, M. A passive bistatic radar experiment for very low radar cross-section target detection. In Proceedings of the Proc. IEEE Radar Conf. 2015; pp. 406–410. [Google Scholar] [CrossRef]
14. Yao, Y.; Zhao, J.; Wu, L. Waveform optimization for target estimation by cognitive radar with multiple antennas. Sensors 2018, 18, 1743. [Google Scholar] [CrossRef] [PubMed]
15. Haykin, S. Cognitive radar: a way of the future. IEEE Signal Process. Mag. 2006, 23, 30–40. [Google Scholar] [CrossRef]
16. Metcalf, J.; Blunt, S.D.; Himed, B. A machine learning approach to cognitive radar detection. In Proceedings of the Proc. IEEE Radar Conf. 2015; pp. 1405–1411. [Google Scholar] [CrossRef]
17. Smith, G.E.; Gurbuz, S.Z.; Brüggenwirth, S.; John-Baptiste, P. Neural Networks & Machine Learning in Cognitive Radar. In Proceedings of the Proc. IEEE Radar Conf. 2020; pp. 1–6. [Google Scholar] [CrossRef]
18. Thornton, C.E.; Kozy, M.A.; Buehrer, R.M.; Martone, A.F.; Sherbondy, K.D. Deep Reinforcement Learning Control for Radar Detection and Tracking in Congested Spectral Environments. IEEE Trans. on Cognitive Commun. and Netw. 2020, 6, 1335–1349. [Google Scholar] [CrossRef]
19. Ahmed, A.M.; Ahmad, A.A.; Fortunati, S.; Sezgin, A.; Greco, M.S.; Gini, F. A Reinforcement Learning Based Approach for Multitarget Detection in Massive MIMO Radar. IEEE Trans. on Aerosp. and Electron. Syst. 2021, 57, 2622–2636. [Google Scholar] [CrossRef]
20. Yang, T.; De Maio, A.; Zheng, J.; Su, T.; Carotenuto, V.; Aubry, A. An Adaptive Radar Signal Processor for UAVs Detection With Super-Resolution Capabilities. IEEE Sensors J. 2021, 21, 20778–20787. [Google Scholar] [CrossRef]
21. Wu, Q.; Wang, H.; Li, X.; Zhang, B.; Peng, J. Reinforcement learning-based anti-jamming in networked UAV radar systems. Appl. Sci. 2019, 9, 5173. [Google Scholar] [CrossRef]
22. Chen, Y.J.; Chang, D.K.; Zhang, C. Autonomous Tracking Using a Swarm of UAVs: A Constrained Multi-Agent Reinforcement Learning Approach. IEEE Trans. on Veh. Technol. 2020, 69, 13702–13717. [Google Scholar] [CrossRef]
23. Mendis, G.J.; Wei, J.; Madanayake, A. Deep learning cognitive radar for micro UAS detection and classification. In Proceedings of the Proc. IEEE Cogn. Commun. for Aerosp. Appl. Workshop (CCAA); 2017; pp. 1–5. [Google Scholar] [CrossRef]