1. Introduction

2. Related works

3. The Proposed Method

3.2. Iterative dual-channel attention fusion (iDAF)

For the iDAF, we design with two data embedding layers to construct the local and global feature maps, then send it into an iterative dual-channel attention module (iDCAM) for attention weight assignment as shown in Figure 1.

3.2.1. Data embedding

The signal data consists of three modal inputs, including I/Q, A/P, and spectrum analysis. For the original signal, it is preprocessed into three modalities inputs, denoted as $h_{IQ}\in {\mathcal{R}}^{128\times 2}$, $h_{AP}\in {\mathcal{R}}^{128\times 2}$, $h_{SP}\in {\mathcal{R}}^{128\times 3}$. The preprocessed inputs $h_m$ ($m\in (IQ,AP,SP)$) represent orthogonal information, amplitude-phase domain, and spectral features respectively.

Due to the variability of multimodal features, direct fusion would ignore the properties unique to different modalities. Therefore, we capture features from both local and global feature maps. The local feature map extracts detailed high-level semantic features, and the global feature map focuses on inter-modal salient characteristics. Therefore, we construct these two feature maps separately using feature extraction networks with different-sized receptive fields.

For the local feature map X, the feature extraction network is expected to focus on local details and contextual information. Inspired by [35], we propose the local embedding layer with CNN, LSTM and DNN which is fine-tuned to extract local attention information. Firstly, preprocessed data passes through a few convolution layers to model frequency. Therefore, the long-term features are obtained by undistorted convolution (UD-Conv) layers with channel dimensions 128, 64, 32, and 16. Concretely, UD-Conv consists of a zero-padding layer of size (2,0,0,0), a convolution layer, the Relu function, and batch normalization. Using the zero-padding, two columns are added to ensure that the signal features can be transmitted with as little time-frequency information as possible. Following [36], the outputs of CNN are sent into LSTM and DNN. The LSTM layer is a bidirectional recursive model with 100 cells, which makes predictions using information both before and after the current moment in the sequence. The input is passed to the model in the original order, the incoming data in the reverse order, and finally the forward and reverse outputs are merged. The long-short time series learning capability of the LSTM identifies temporal correlations in I/Q data with inherent memory properties, and benefits learning the temporal dependencies of instantaneous amplitude and phase [15].

Figure 2. The IQ data plot of 11 modulations.

Figure 2. The IQ data plot of 11 modulations.

Figure 3. The AP data plot of 11 modulations.

Figure 3. The AP data plot of 11 modulations.

Figure 4. The SP data plot of 11 modulations.

Figure 4. The SP data plot of 11 modulations.

The residual mapping function is a shortcut path between different layers, which can deepen the communication between deep and shallow neural network features. Inspired by [14], the Resnet has achieved the best performance on classifying signal modulation with a 4-convolution-layer structure. After four UD-Conv layers, long-term features are extracted by the convolution layers, while short-term information may be neglected during the convolution process. Therefore, the original data containing long-short-term features are entered into LSTM together with the extracted long-term features via the residual connection. Inspired by [37], the extracting capability of CNN is combined with LSTM and DNN. As shown in Figure 5a, the learned short-term features are fed into the dense layer together with the long-term features previously extracted by CNN. The local embedding layer captures the data characteristics of each modal with unshared parameters, which is expressed as $x_m=E_x\left(h_m,{\omega }_m^{E_x}\right),(m\in (IQ,AP,SA))$, where $E_x$ represents the local embedding layer and ${\omega }_m$ indicates the local network parameters.

To obtain the global feature map Y, an optimized CNN with 3 convolutional layers is utilized to extract features $y_m=E_y\left(h_m,{\omega }_m^{E_y}\right),(m\in (IQ,AP,SA))$ in the global receptive field in Figure 5b.

3.2.2. Dual-scale channel attention module

After constructing the feature maps in the previous section, the feature maps are fed into an iterative multimodal attention module (iDCAM). The dual-scale channel attention module (DCAM) is a computational unit that can be constructed and superimposed for feature map transformation containing two branches.

Figure 6. Architecture of the proposed Dual-scale channel attention module (DCAM).

Figure 6. Architecture of the proposed Dual-scale channel attention module (DCAM).

The branches include a local attention branch and a global attention branch, correspondingly for extracting the local identification properties and the channel variability between modalities, respectively. The local attention branch extracts the intra-modal attention through the self-attention mechanism of the Transformer, which extracts the local recognition properties of specific modality features. Meanwhile, the global attention branch increases the receptive field by pooling to obtain inter-modal global attention in the channel dimension. The feature maps are respectively fed into the dual-scale channel attention module and the following steps are performed as follows:

1) Passing through the encoder.

To capture the attention information between different modalities, the feature map is first sent into the encoder layer of the Transformer [38]. The encoder consists of a self-attention module and a feed-forward neural network. Concretely, the self-attention mechanism is able to interact with the vectors converted from different sequence tokens, giving attention information about the correlation between different modalities. The basic formula of the self-attention mechanism is first expressed as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}Q& =W_{Q}x\\ K& =W_{K}x\\ V& =W_{V}x\end{array}\right.\end{array}$$

(5)

Therefore, the input x is converted to a query Q, a key K, and a value V by means of three learnable weights $W_Q$,$W_K$, and $W_V$. Here, Q is used to query the similarity of other vectors to itself and K is used for indexing for operations.

By dot-multiplying Q and K, the similarity between the two is computed, which is then converted into a weight probability distribution to get the importance of different modalities in different signal sequences as attention information. Specifically, the attention information is normalized by scaling factor and softmax.

$$\begin{array}{c}\hfill Atten(Q,K,V)=softmax\left(\frac{Q{K}_T}{\sqrt{d_k}}\right)V\end{array}$$

(6)

Finally, the output of this self-attention layer is obtained by weighting the value V which helps in the classification with the attention information and then accumulating it. Utilizing multiple self-attention layer operations, the multi-head attention layer as shown in the following equation:

$$\begin{array}{cc}\hfill G\left(X\right)=Multihead(Q,K,V)& =concat(hea{d}_1,hea{d}_2,hea{d}_n)W\hfill \end{array}$$

(7)

$$$$

(8)

$$\begin{array}{cc}\hfill hea{d}_n& =Atten(Q_n,K_n,V_n)s_{n}ftmax\left(\frac{Q_n{K}_n^T}{\sqrt{d_k}}\right)V_n\hfill \end{array}$$

(9)

2) Construct the global channel attention matrix.

First, feature mappings across spatial dimensions H × W are aggregated after a squeeze compression operation. A channel descriptor containing global attention information is generated by global average pooling, which is denoted as follows:

$$\begin{array}{c}\hfill z_i=F_{squ}\left(x_i\right)=\frac{1}{H\times W}\sum _{m=1}^H\sum _{n=1}^H{x}_i(m,n)\end{array}$$

(10)

After squeeze compression, the aggregated information is sent into two convolution layers to capture the channel dependencies.

$$\begin{array}{c}\hfill L\left(X\right)=F_{conv}\left(z\right)=B\left(Con{v}_1\left(\sigma \left(B\left(Con{v}_2\left(z\right)\right)\right)\right)\right)\end{array}$$

(11)

where $\sigma$ and B represent the Rectified Linear Unit (ReLU) function and Batch Normalization (BN), respectively. Specifically, the $Conv$ we used is the point-wise convolution, which enhances the nonlinear capabilities of the network. The kernel size of $Con{v}_1$ and $Con{v}_1$ is $1\times 1\times 1$ and $3\times 1\times 1$ respectively.

3) Matrix multiplication between the attention matrix and the original features.

$$\begin{array}{c}\hfill H^{\prime }=H\otimes W((X\oplus Y)=H\otimes \sigma (L\left(X\right)\oplus G\left(X\right)),\end{array}$$

(12)

where ⊗ represents matrix addition and ⊕ is matrix multiplication. $W\left(X\right)$ contains the summation information of local attention $L\left(X\right)$ and global attention $G\left(X\right)$ extracted through DCAM.

3.2.3. Iteritive dual-scale attentional module

The inputs are high-level feature maps X and low-level feature maps Y. X utilizes the local sensing and context-sensitive inference capabilities of CNN and LSTM to capture the discriminative properties of each modality. However, the extracted high-level features are rich in local semantic information but ignore inter-modal difference information. In contrast, Y extracts global information with a larger perceptual field, and the extracted low-level features extract the distinctiveness between different modalities from a holistic perspective. However, due to the use of fewer convolutional layers, the deep feature semantic information is difficult to mine. Therefore, due to the desire to complement the advantages of low-level features and high-level features, an iterative dual-scale attentional module (iDCAM) is designed.

Figure 7. The iterative attention mechanism iDCAM.

Figure 7. The iterative attention mechanism iDCAM.

By stacking the DCAM designed in the previous section, iDCAM assigns multimodal attention weights to different modality features.

$$\begin{array}{c}\hfill Z_i=W_i(X\oplus Y)\otimes X+(1-W_i(X\oplus Y)\otimes X,i=1,2,...k\end{array}$$

(13)

where $W(X\oplus Y)$ represents the summation information of local X and global Y.

3.2.4. Residual encoder

After passing through iDCAM, features are sent into the decoder along with the sum of intermediate features. The features are fed into the decoder after being assigned weights by iDCAM, and decoding is guided by a cross-attention mechanism using the sum of the intermediate features $x_m$ ($m\in (IQ,AP,SA)$) and $y_m$ ($m\in (IQ,AP,SA)$).

4. Experiment Results and Discussion

5. Conclusion

References

1. Dai, A.; Zhang, H.; Sun, H. Automatic modulation classification using stacked sparse auto-encoders. In Proceedings of the 2016 IEEE 13th international conference on signal processing (ICSP). IEEE; 2016; pp. 248–252. [Google Scholar]
2. Al-Nuaimi, D.H.; Hashim, I.A.; Zainal Abidin, I.S.; Salman, L.B.; Mat Isa, N.A. Performance of feature-based techniques for automatic digital modulation recognition and classification—A review. Electronics 2019, 8, 1407. [Google Scholar] [CrossRef]
3. Bhatti, F.A.; Khan, M.J.; Selim, A.; Paisana, F. Shared spectrum monitoring using deep learning. IEEE Transactions on Cognitive Communications and Networking 2021, 7, 1171–1185. [Google Scholar] [CrossRef]
4. Richard, G.; Wiley, E. The interception and analysis of radar signals. Artech House, Boston 2006.
5. Kim, K.; Spooner, C.M.; Akbar, I.; Reed, J.H. Specific emitter identification for cognitive radio with application to IEEE 802.11. In Proceedings of the IEEE GLOBECOM 2008-2008 IEEE Global Telecommunications Conference. IEEE, 2008, pp. 1–5.
6. Wei, W.; Mendel, J.M. Maximum-likelihood classification for digital amplitude-phase modulations. IEEE transactions on Communications 2000, 48, 189–193. [Google Scholar] [CrossRef]
7. Xu, J.L.; Su, W.; Zhou, M. Likelihood-ratio approaches to automatic modulation classification. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 2010, 41, 455–469. [Google Scholar] [CrossRef]
8. Hazza, A.; Shoaib, M.; Alshebeili, S.A.; Fahad, A. An overview of feature-based methods for digital modulation classification. In Proceedings of the 2013 1st international conference on communications, signal processing, and their applications (ICCSPA). IEEE; 2013; pp. 1–6. [Google Scholar]
9. Hao, Y.; Wang, X.; Lan, X. Frequency Domain Analysis and Convolutional Neural Network Based Modulation Signal Classification Method in OFDM System. In Proceedings of the 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP). IEEE; 2021; pp. 1–5. [Google Scholar]
10. Ali, A.; Yangyu, F. Unsupervised feature learning and automatic modulation classification using deep learning model. Physical Communication 2017, 25, 75–84. [Google Scholar] [CrossRef]
11. Chang, S.; Huang, S.; Zhang, R.; Feng, Z.; Liu, L. Multitask-learning-based deep neural network for automatic modulation classification. IEEE internet of things journal 2021, 9, 2192–2206. [Google Scholar] [CrossRef]
12. O’Shea, T.J.; Corgan, J.; Clancy, T.C. Convolutional radio modulation recognition networks. In Proceedings of the Engineering Applications of Neural Networks: 17th International Conference, EANN 2016, Aberdeen, UK, September 2-5, 2016, Proceedings 17. Springer, 2016, pp. 213–226.
13. Ke, Z.; Vikalo, H. Real-time radio technology and modulation classification via an LSTM auto-encoder. IEEE Transactions on Wireless Communications 2021, 21, 370–382. [Google Scholar] [CrossRef]
14. Liu, X.; Yang, D.; El Gamal, A. Deep neural network architectures for modulation classification. In Proceedings of the 2017 51st Asilomar Conference on Signals, Systems, and Computers. IEEE; 2017; pp. 915–919. [Google Scholar]
15. Rajendran, S.; Meert, W.; Giustiniano, D.; Lenders, V.; Pollin, S. Deep learning models for wireless signal classification with distributed low-cost spectrum sensors. IEEE Transactions on Cognitive Communications and Networking 2018, 4, 433–445. [Google Scholar] [CrossRef]
16. Zhang, Z.; Wang, C.; Gan, C.; Sun, S.; Wang, M. Automatic modulation classification using convolutional neural network with features fusion of SPWVD and BJD. IEEE Transactions on Signal and Information Processing over Networks 2019, 5, 469–478. [Google Scholar] [CrossRef]
17. Zeng, Y.; Zhang, M.; Han, F.; Gong, Y.; Zhang, J. Spectrum analysis and convolutional neural network for automatic modulation recognition. IEEE Wireless Communications Letters 2019, 8, 929–932. [Google Scholar] [CrossRef]
18. Shi, F.; Hu, Z.; Yue, C.; Shen, Z. Combining neural networks for modulation recognition. Digital Signal Processing 2022, 120, 103264. [Google Scholar] [CrossRef]
19. Fu-qing, H.; Zhi-ming, Z.; Yi-tao, X.; Guo-chun, R. Modulation recognition of symbol shaped digital signals. In Proceedings of the 2008 International Conference on Communications, Circuits and Systems. IEEE; 2008; pp. 328–332. [Google Scholar]
20. Zhang, X.; Li, T.; Gong, P.; Liu, R.; Zha, X. Modulation recognition of communication signals based on multimodal feature fusion. Sensors 2022, 22, 6539. [Google Scholar] [CrossRef] [PubMed]
21. Qi, P.; Zhou, X.; Zheng, S.; Li, Z. Automatic modulation classification based on deep residual networks with multimodal information. IEEE Transactions on Cognitive Communications and Networking 2020, 7, 21–33. [Google Scholar] [CrossRef]
22. Zhang, Z.; Luo, H.; Wang, C.; Gan, C.; Xiang, Y. Automatic modulation classification using CNN-LSTM based dual-stream structure. IEEE Transactions on Vehicular Technology 2020, 69, 13521–13531. [Google Scholar] [CrossRef]
23. Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
24. Hu, J.; Shen, L.; Albanie, S.; Sun, G.; Vedaldi, A. Gather-excite: Exploiting feature context in convolutional neural networks. Advances in neural information processing systems 2018, 31. [Google Scholar]
25. Gao, Z.; Xie, J.; Wang, Q.; Li, P. Global second-order pooling convolutional networks. In Proceedings of the Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2019, pp. 3024–3033.
26. Lee, H.; Kim, H.E.; Nam, H. Srm: A style-based recalibration module for convolutional neural networks. In Proceedings of the Proceedings of the IEEE/CVF International conference on computer vision; 2019; pp. 1854–1862. [Google Scholar]
27. Mnih, V.; Heess, N.; Graves, A.; et al. Recurrent models of visual attention. Advances in neural information processing systems 2014, 27. [Google Scholar]
28. Wang, X.; Girshick, R.; Gupta, A.; He, K. Non-local neural networks. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition; 2018; pp. 7794–7803. [Google Scholar]
29. Yang, F.; Yang, L.; Wang, D.; Qi, P.; Wang, H. Method of modulation recognition based on combination algorithm of K-means clustering and grading training SVM. China communications 2018, 15, 55–63. [Google Scholar]
30. Hussain, A.; Sohail, M.; Alam, S.; Ghauri, S.A.; Qureshi, I.M. Classification of M-QAM and M-PSK signals using genetic programming (GP). Neural Computing and Applications 2019, 31, 6141–6149. [Google Scholar] [CrossRef]
31. Das, D.; Bora, P.K.; Bhattacharjee, R. Blind modulation recognition of the lower order PSK signals under the MIMO keyhole channel. IEEE Communications Letters 2018, 22, 1834–1837. [Google Scholar] [CrossRef]
32. Liu, Y.; Liang, G.; Xu, X.; Li, X. The Methods of Recognition for Common Used M-ary Digital Modulations. In Proceedings of the 2008 4th International Conference on Wireless Communications, Networking and Mobile Computing; 2008; pp. 1–4. [Google Scholar] [CrossRef]
33. Benedetto, F.; Tedeschi, A.; Giunta, G. Automatic Blind Modulation Recognition of Analog and Digital Signals in Cognitive Radios. In Proceedings of the 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall); 2016; pp. 1–5. [Google Scholar] [CrossRef]
34. Jiang, K.; Zhang, J.; Wu, H.; Wang, A.; Iwahori, Y. A novel digital modulation recognition algorithm based on deep convolutional neural network. Applied Sciences 2020, 10, 1166. [Google Scholar] [CrossRef]
35. Sainath, T.N.; Vinyals, O.; Senior, A.; Sak, H. Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2015; pp. 4580–4584. [Google Scholar] [CrossRef]
36. Sermanet, P.; LeCun, Y. Traffic sign recognition with multi-scale convolutional networks. In Proceedings of the The 2011 international joint conference on neural networks. IEEE; 2011; pp. 2809–2813. [Google Scholar]
37. Soltau, H.; Saon, G.; Sainath, T.N. Joint training of convolutional and non-convolutional neural networks. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2014; pp. 5572–5576. [Google Scholar]
38. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N. Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30. [Google Scholar]
39. Park, J.; Woo, S.; Lee, J.Y.; Kweon, I.S. Bam: Bottleneck attention module. arXiv 2018, arXiv:1807.06514 2018. [Google Scholar]
40. Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
41. Li, X.; Wang, W.; Hu, X.; Yang, J. Selective kernel networks. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 510–519.