Adaptive GCN and Bi-GRU Based Dual-Branch for Motor lmagery EEG Decoding

1. Introduction

Motor Imagery Brain-Computer Interface (MI-BCI) enables direct interaction between the brain and external devices by decoding electrical activity from the central nervous system [1,2,3]. This technology is valuable in medical rehabilitation [4] and stroke treatment [5]. However, electroencephalogram (EEG) signals are characterized by low signal-to-noise ratios [6], non-stationarity [7], and significant inter-individual variability, posing substantial challenges for decoding EEG signals [8].

Machine learning techniques have been widely applied in the decoding of Motor Imagery Electroencephalogram (Ml-EEG) signals.The Common Spatial Pattern (CSP) is a commonly used method for spatial feature extraction [9], which enhances the separability of signals by maximizing the variance between classes. The Filter, Bank CSP (FBCSP) overcomes the frequency-band dependence issue of CSP [10]. Mojgan et al. [11] adopted root-mean-square and waveform length as temporal features, achieving better performance than CSP. Liu et al. [12] reached an accuracy 93.31% by using Power Spectral Density (PSD) features. Zhang et al. [13] decomposed EEG signals into sub-bands with wavelet packets, extracted temporal-spectral features through an autoregressive model, and classified them using a Support Vector Machine (SVM). Yang et al. [14] utilized Riemann covariance across multiple frequency bands and achieved an accuracy 76%. Despite these advancements, machine learning methods rely on manual feature extraction and are unable to reveal deep latent features, thus limiting the decoding performance of MI-EEG.

With the rapid development of deep learning, many studies in recent years have applied deep learning methods such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs) [15], and recurrent neural networks (RNNs) [16] to EEG signals, achieving good decoding results. Alhagry et al. [17] proposed using the LSTM method to study the time-varying characteristics of EEG. Li et al. [18] proposed a bidirectional ConvLSTM model applied to EEG to explore deep temporal features. Schirrmeister et al. [19] proposed the Shallow ConvNet model, Lawhern VJ et al. [20] proposed the EEGNet model, and models such as Multi-branch-3D [21] and MSFBCNN [22] achieved good classification results in an end-to-end manner without requiring prior knowledge .

The above deep learning methods based on convolutional and recurrent units can easily obtain hidden features through network training. However, when subjects perform MI tasks, the activation intensity of different brain regions varies significantly, and the spatial distribution of EEG signals is irregular [23], making CNNs unable to handle the connectivity between MI-EEG channels [24,25]. In recent years, graph convolutional networks (GCNs) have been proven to capture the relationships between channels using adjacency matrices [26,27,28]. Zhang et al. [29] proposed the GCB-Net generalized graph convolutional network, extracting deeper spatial-spectral information. Song et al. [30] proposed a dynamic graph convolutional neural network that dynamically learns the graph structure of EEG signals through network training. Chen et al. [31] combined the self-attention mechanism and spatial graph convolution to fully utilize the correlations between EEG channels, constructing a classification model that obtained richer spatial features .

The channel connectivity, temporal dependency, and frequency domain information of MI-EEG are all crucial for improving decoding performance. This is because the signals induced by motor imagery in the sensory-motor cortex exhibit event-related desynchronization/synchronization (ERD/ERS) at specific rhythms, brain regions, and time segments. To fully exploit multi-domain features, some researchers have proposed feature fusion methods. Tang et al. [32] proposed the SF-TGCN model, utilizing multi-layer temporal convolution modules and the Laplacian operator to achieve deep temporal and spatial feature fusion of MI-EEG. Li et al. [33] utilized temporal convolution blocks and multi-level wavelet convolution to achieve temporal-frequency feature fusion, achieving an accuracy of 74.71% in four-class motor imagery classification tasks .

However, existing studies only aggregate information from two dimensions, without fully utilizing temporal, frequency, and spatial information simultaneously. Therefore, this paper proposes a dual-branch network to simultaneously explore the channel connectivity, temporal dependency, and frequency domain information of MI-EEG: one branch uses Adaptive GCN to establish the connectivity between channels, capturing the synchronization state and dynamic process of the brain, and employs CBAM to focus on important features, achieving spatial-spectral feature extraction. The other branch employs temporal convolution to serialize the MI-EEG, utilizing Bi-GRU and multi-head self-attention mechanisms to extract deep temporal-spectral features from the dependency relationships in the sequence. Feature fusion is then performed. The main contributions of our work are:

• An adaptive graph convolutional network is proposed, which constructs the graph convolutional layers of MI-EEG signals using a dynamic adjacency matrix. The CBAM is employed to focus on important features, capturing the synchronized activity states and dynamic processes of the brain, thereby improving the quality of spatial-temporal feature decoding.
• Propose a time-frequency feature extraction model to fully explore the sequential dependencies and global dependencies among features of different time segments. This study combines the bidirectional gated recurrent unit (Bi-GRU) with the multi-head self-attention mechanism to obtain deep-level time-frequency features.
• Experiments on multiple public EEG datasets show that, compared with state-of-the-art methods, the proposed model achieves significant improvements, enhancing the decoding quality of MI-EEG multi-classification tasks.

The rest of this paper is organized as follows. Section 2 describes the proposed network structure. Section 3 details the experimental results. Section 4 discusses the effect of the proposed method on the utilization of channel correlation and temporal dependence. Finally, the paper is summarized in Section 5.

2. Materials and Methods

2.1. Dataset

This study utilized the BCI-IV Dataset 2a and BCI-III Dataset 3a, which involve four categories of body movement imagery tasks: left-hand, right-hand, both feet, and tongue.

The BCI-IV Dataset 2a comprises EEG data from nine subjects (A01-A09), with each subject performing 576 motor imagery tasks. Each task lasted approximately 8 seconds, including a 2-second task warning period, a 4-second motor imagery period, and a 2-second rest period. EEG signals were recorded from twenty-two channels at a sampling frequency of 250 Hz. Post-acquisition, bandpass filtering from 0.5 to 100 Hz and 50 Hz notch filtering were applied (Figure 1).

The BCI-III Dataset 3a includes EEG data from three participants (K3b, K6b, L1b), with K3b having 360 trial sets and the other two participants having 240 trial sets each. Each imagery task lasted 4 seconds, with EEG signals recorded from 60 channels at a sampling frequency of 250 Hz. To align with the channel selection in BCI-IV Dataset 2a, 22 electrodes near the central scalp region were selected.

2.2. Proposed Model Framework

This work proposes a dual-branch network using Adaptive GCN and Bi-GRU to enhance MI-EEG decoding performance by simultaneously mining channel correlation and temporal dependency. The model architecture, shown in Figure 2, includes preprocessing, spatio-spectral feature extraction, temporal-spectral feature extraction, and feature fusion. In preprocessing, the original signal is segmented into nine sub-bands, with spatial filtering applied to each to obtain ERD/ERS modes across various frequency bands. Branch I utilizes Adaptive GCN and CBAM to extract spatial-spectral features from EEG channel functional connectivity, leveraging an adaptive adjacency matrix to capture synchronous brain activity states and dynamic processes. Branch II employs Bi-GRU and MHA to extract temporal-spectral features by capturing temporal dependencies in multi-band EEG sequences. Finally, a feature layer fusion strategy is adopted to integrate diverse information, which is then sent to the Softmax layer for classification.

2.3. Preprocessing

Distinguishing motor imagery actions can be achieved through event-related synchronization (ERS) and desynchronization (ERD) [34], which are prominent in the $\alpha$ (8∼13 Hz) and $\beta$ (14∼30 Hz) bands. To preserve ERD/ERS modes at different frequencies, subband division and CSP-based spatial filtering are applied to the original MI-EEG, inspired by the FBCSP [35] algorithm.

A filter bank was constructed using nine overlapping Chebyshev Type II bandpass filters, each with a 6 Hz bandwidth, covering a range of 4∼42 Hz (4∼10, 8∼14, ..., 36∼42 Hz). Each sub-band is spatially filtered as follows:

$${\mathit{Z}}_{b,i}={\overline{\mathit{W}}}_b^{\mathrm{T}}{\mathit{E}}_{b,i}$$

(1)

where${\mathit{Z}}_{b,i}$ is the signal after spatial filtering, ${\mathit{E}}_{b,i}\in {\mathbb{R}}^{c\times t}$ represents the i-th EEG sample in the b-th frequency band, c is the number of EEG channels, t denotes the number of sampling points per channel, and T is the transpose. ${\overline{\mathit{W}}}_b$ represents the CSP projection matrix derived from the eigenvalue decomposition of Eq. 2:

$${\Sigma }_{b,1}{\mathit{W}}_b=({\Sigma }_{b,1}+{\Sigma }_{b,2}){\mathit{W}}_b{\mathit{D}}_b$$

(2)

where ${\Sigma }_{b,1}$ and ${\Sigma }_{b,2}$ represent the covariance matrix estimates for the first and second classes of MI tasks in the b-th frequency band, respectively, ${\mathit{D}}_b$ is a diagonal array consisting of the eigenvalues of ${\Sigma }_{b,1}$. The first n and the last n column vectors from ${\mathit{W}}_b\in {\mathbb{R}}^{c\times c}$ are selected to form the spatial domain filter ${\overline{\mathit{W}}}_b$. For the four-class MI tasks, a One-vs-Rest strategy is adopted. One MI task is considered the first class, while the other tasks are combined into the second class. The final spatial filter ${\overline{\mathit{W}}}_b$ contains $k\times 2\times n$ vectors, where $k=4$ represents the four classifications, and $n=2$ is set. After spatial filtering according to Eq. 1, the number of EEG channels is 16. After spatial filtering of the 9 band signals, the dimension of the input data ${\mathit{f}}_{in}$ becomes $9\times 1000\times 16$.

2.4. Spatio-Spectral Feature Extraction Module

EEG exhibits complex dynamic functional connections between different brain regions. Graph convolutional networks have been proven to capture the correlation between channels effectively. Therefore, a spatio-spectral feature extraction module is designed to capture EEG channel correlations across frequency bands and fully extract spatio-spectral features.

The topology of MI-EEG must first be constructed to process EEG signals using graph convolution [36]. An undirected EEG signal connectivity graph is denoted as $\mathit{G}=\{\mathit{V},\mathit{E},\mathit{A}\}$, where $\mathit{V}=\{{\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_i,\dots {\mathit{v}}_N\}$ represents the set of nodes, ${\mathit{v}}_i$ denotes the electrode channel, $\mathit{E}$ is the set of edges connecting the nodes, and $(\mathit{v}i,\mathit{v}j)\in$$\mathit{E}$, $\mathit{A}=\{a_{ij}\in {\mathit{R}}^{N\times N};i,j=1,\dots ,N\}$ represents the adjacency matrix describing the connection strength between any two nodes. In constructing A, the traditional approach considers the electrode distribution as a non-Euclidean space, representing the spatial disorder of the local EEG channel by the adjacency of electrode nodes. Since the ERD/ERS phenomenon in MI-EEG is distributed across multiple brain regions, electrode interactions must be reflected globally. In this paper, A is constructed using the Pearson correlation coefficient P, as in Eq. 3, which captures synchronous brain activity information and retains the negative correlation of EEG channel activity by taking the absolute value of P. Additionally, introducing the self-connectivity coefficient $\alpha$ enhances the weight of the self-node.

$$\mathit{A}=\mid \mathit{P}\mid +\alpha \mathit{I}$$

(3)

Since motor imagery is a dynamic process, Adaptive GCN [37] was used in constructing MI-EEG graphs to capture dynamically changing brain connectivity states. This method allows for adaptive adjustment of graph connection weights and enhances the extraction of deep spatio-spectral features.

Specifically, the spatio-spectral features ${\tilde{\mathit{f}}}_{sf}$ obtained by Adaptive GCN are computed as follows:

$${\tilde{\mathit{f}}}_{sf}={\mathit{W}}_{1\times 1}*({\tilde{\mathit{D}}}^{-1/2}\mathit{A}{\tilde{\mathit{D}}}^{-1/2}+\mathit{B}+\mathit{C}){\mathit{f}}_{in}$$

(4)

The adjacency matrix $\mathit{A}\in {\mathbb{R}}^{N\times N}$ is calculated using Eq. 3, $\tilde{\mathit{D}}$ is the degree matrix used to standardize $\mathit{A}$. $\mathit{B}\in {\mathbb{R}}^{N\times N}$ represents the shared adjacency matrix for all samples, retaining common information. It is a trainable parameter that is dynamically updated during model training. The update process follows Eq. 5, with r as the learning rate and $Loss$ as the model training loss.

$$\mathit{B}=(1-r)\mathit{B}+r{\displaystyle \frac{\partial Loss}{\partial \mathit{B}}}$$

(5)

$\mathit{C}\in {\mathbb{R}}^{bs\times N\times N}$ represents the adjacency matrix associated with the sample data, assigning unique connection strengths to each input data, computed as in Eq. 6, ${\mathit{W}}_{\theta }$ and ${\mathit{W}}_{\varphi }$ denote two 1*1 convolution kernels.

$$\mathit{C}=softmax\left({\mathit{f}}_{in}^T{\mathit{W}}_{\theta }^T{\mathit{W}}_{\varphi }{\mathit{f}}_{in}\right)$$

(6)

Redundant information exists in the spatio-spectral features ${\tilde{\mathit{f}}}_{sf}$ extracted by Adaptive GCN. Therefore, this paper introduces CBAM [38] (Figure 3), which uses an attention mechanism in both feature space and feature channel dimensions to suppress redundant features and improve the quality of the spatio-spectral features.

Feature channel attention emphasizes important feature channels. The spatial dimensionality of ${\tilde{\mathit{f}}}_{sf}$ is compressed using average and maximum pooling to generate the matrices ${\mathit{F}}_{avg}^{\mathit{c}}$ and ${\mathit{F}}_{max}^{\mathit{c}}$, respectively. A multilayer perceptron (MLP) then generates the channel attention mapping ${\mathit{M}}_{\mathit{c}}$:

$${\mathit{M}}_{\mathit{c}}\left(\mathit{F}\right)=\sigma ({\mathit{W}}_1\left({\mathit{W}}_0\left({\mathit{F}}_{avg}^{\mathit{c}}\right)\right)+{\mathit{W}}_1\left({\mathit{W}}_0\left({\mathit{F}}_{max}^{\mathit{c}}\right)\right))$$

(7)

Where $\sigma$ represents the sigmoid function, ${\mathit{W}}_0$ and ${\mathit{W}}_1$ are the weights of MLP. Feature space attention emphasizes the target region of interest, complementing channel attention. It is computed as in Eq. 8 by applying average and maximum pooling along the feature channel dimensions to obtain the matrices ${\mathit{F}}_{avg}^{\mathit{s}}\in {\mathcal{R}}^{1\times 1000\times 16}$ and ${\mathit{F}}_{max}^{\mathit{s}}\in {\mathcal{R}}^{1\times 1000\times 16}$. These matrices are concatenated and then convolved with a $3\times 3$ kernel to generate ${\mathit{M}}_s$.

$$\begin{array}{cc}\hfill {\mathit{M}}_s\left(F\right)& =\sigma \left(f^{3\times 3}\left([AvgPool\left({\mathit{f}}_{out}^{\prime }\right);MaxPool\left({\mathit{f}}_{out}^{\prime }\right)]\right)\right)\hfill \\ \hfill & =\sigma \left(f^{3\times 3}\left([{\mathit{F}}_{a\nu g}^s;{\mathit{F}}_{max}^s]\right)\right)\hfill \end{array}$$

(8)

The feature ${\tilde{\mathit{f}}}_{sf}$ is element-wise multiplied with ${\mathit{M}}_c\in R^{8\times 1\times 1}$ and ${\mathit{M}}_s\in R^{1\times 1000\times 16}$ to obtain the weighted spatio-spectral feature ${\mathit{f}}_{sf}$.

$$\begin{array}{c}\hfill {\mathit{f}}_{out}^{\prime }=M_c\left({\tilde{\mathit{f}}}_{sf}\right)\otimes {\tilde{\mathit{f}}}_{sf}\\ \hfill {\mathit{f}}_{sf}=M_s\left({\mathit{f}}_{out}^{\prime }\right)\otimes {\mathit{f}}_{out}^{\prime }\end{array}$$

(9)

Figure 4 presents the implementation details of the AGCN layer, which combines the AGCN layer, batch normalization (BN), and the ReLU activation function to form a foundational graph convolutional network (GCN) module. Multiple GCN modules are then stacked to construct the complete graph convolutional network.

2.5. Temproal-Spectral Feature Fxtraction Module

MI-EEG is a highly time-varying signal, dependent on preceding and succeeding time segments [39]. To fully leverage the temporal dependencies and enhance MI-EEG decoding performance, this paper proposes a temporal-frequency feature extraction module, comprising a temporal convolution layer, a single-layer Bi-GRU, and an MHA module. The temporal convolution layer consists of a two-dimensional convolution filter of size (50, 1). The input is ${\mathit{f}}_{in}\in {\mathbb{R}}^{9\times 1000\times 16}$, and the output is the temporal-spectral feature $\mathit{x}\in {\mathbb{R}}^{16\times 191}=[{\mathit{x}}_1,...,{\mathit{x}}_t,...,{\mathit{x}}_{191}]$. Each sequence ${\mathit{x}}_t$ in $\mathit{x}$ aggregates the temporal-spectral information of a 0.2s time window (the sampling frequency is 250Hz, and 50 sampling points are recorded in 0.2s). The Bi-GRU [40] model captures the forward and backward dependencies of the sequence ${\mathit{x}}_t$. Bi-GRU, a type of RNN with a gating structure, includes an update gate ${\mathit{z}}_t$ and a reset gate ${\mathit{r}}_t$. This gating structure selectively transfers information in the hidden layer, solving the gradient vanishing problem in RNNs and overcoming short-term memory issues, as calculated by the following formulas:

$${\mathit{r}}_t=\sigma ({\mathit{W}}_r{\mathit{x}}_t+{\mathit{U}}_r{\mathit{h}}_{t-1}+{\mathit{b}}_r)$$

(10)

$${\mathit{z}}_t=\sigma ({\mathit{W}}_z{\mathit{x}}_t+{\mathit{U}}_z{\mathit{h}}_{t-1}+{\mathit{b}}_z)$$

(11)

$\mathit{W}$, $\mathit{U}$, $\mathit{b}$ are the weight parameter matrices and bias, respectively. ${\mathit{h}}_{t-1}$ represents the hidden state of the $(t-1)th$ temporal-spectral sequence containing all information before the $(t-1)$ time slice. $\sigma$ is the sigmoid nonlinear activation function. Using the update gate ${\mathit{z}}_t$ and reset gate ${\mathit{r}}_t$, the hidden layer state information ${\overrightarrow{\mathit{h}}}_t$ of the current sequence ${\mathit{x}}_t$ is computed as shown in Eq. 12, ${\tilde{\mathit{h}}}_t$ represents the candidate hidden state, and ⊙ denotes element-wise multiplication.

$$\begin{array}{cc}\hfill & {\tilde{\mathit{h}}}_t=tanh({\mathit{W}}_h{\mathit{x}}_t+{\mathit{U}}_h({\mathit{r}}_t\odot {\mathit{h}}_{t-1})+{\mathit{b}}_h)\hfill \\ \hfill & {\overrightarrow{\mathit{h}}}_t=(1-{\mathit{z}}_t)\odot {\mathit{h}}_{t-1}+{\mathit{z}}_t\odot {\tilde{\mathit{h}}}_t\hfill \end{array}$$

(12)

Bi-GRU computes the hidden states ${\overrightarrow{\mathit{h}}}_t$ and ${\overleftarrow{\mathit{h}}}_t$ from the forward and backward directions, and concatenates them to obtain the temporal-spectral feature ${\overleftrightarrow{\mathit{h}}}_t$ , which contains both forward and backward dependencies. To improve the model’s capacity to capture long-range dependencies within the sequences, the MHA [41] is employed to capture global dependencies among ${\overleftrightarrow{\mathit{h}}}_t$. The MHA module performs self-attention computations with multiple attention heads and outputs the final temporal-spectral feature ${\mathit{f}}_{tf}$ following a linear transformation, ${\mathit{W}}_O$ represents the weight matrix.

$${\mathit{f}}_{tf}=[{\mathit{head}}_1;...;{\mathit{head}}_i;...;{\mathit{head}}_h]{\mathit{W}}_O$$

(13)

$${\mathit{head}}_i=softmax\left({\displaystyle \frac{{\mathit{QW}}_Q^i·{\left({\mathit{KW}}_K^i\right)}^T}{\sqrt{d_k}}}\right){\mathit{VW}}_V^i$$

(14)

Each attention head is computed as in Eq. 14, $\mathit{Q}$, $\mathit{K}$, $\mathit{V}$ are the matrices obtained by linear transformation of ${\overleftrightarrow{\mathit{h}}}_t$, ${\mathit{W}}_Q^i$, ${\mathit{W}}_K^i$, ${\mathit{W}}_V^i$ represent the trainable weight matrices, and $d_k=16$ is the feature dimension of the sequence ${\overleftrightarrow{\mathit{h}}}_t$.

2.6. Feature Fusion

The extracted spatio-spectral features ${\mathit{f}}_{sf}$ and temporal-spectral features ${\mathit{f}}_{tf}$ are fused. The features ${\mathit{f}}_{sf}$ and ${\mathit{f}}_{tf}$ are reduced to 8 and 16 dimensions using average aggregation. The spliced features are linearly transformed using a single-layer neural network, as shown in Eq. 15, with $\mathit{W}$ and $\mathit{b}$ as the weights and bias. The features are converted into probabilities using the Softmax function in Eq. 16 for normalization and dividing EEG into four categories.

$${\mathit{x}}^{\prime }={\mathit{W}}^{4\times 24}\left([\mathrm{Avgagg}\left({\mathit{f}}_{sf}\right),\mathrm{Avgagg}\left({\mathit{f}}_{tf}\right)]\right)+{\mathit{b}}^{4\times 1}$$

(15)

$${\mathit{x}}_{out}^{\prime }\left(i\right)={\displaystyle \frac{exp\left({\mathit{x}}^{\prime }\left(i\right)\right)}{{\sum }_{k=1}^{4}exp\left({\mathit{x}}^{\prime }\left(k\right)\right)}},i=1,...,4$$

(16)

3. Results

3.1. Software and Hardware Environment

The models were implemented in a Python 3.9 environment using the PyTorch 1.13.1 framework and executed on a GeForce 3050 GPU. Model parameter settings are detailed in Table 1. The final results were derived from the average of five separate experiments, using an 8:2 split ratio for training and test sets.

3.2. Classification Results

The classification performance of our proposed dual-branch network across subjects is summarized in Table 2. On average, the model achieved an accuracy of 82.16% and a Kappa value of 0.761. Notably, accuracy exceeded 90% for three participants. However, subjects A02 and A06 showed significantly lower accuracies, below 70%, indicating poor classification. To investigate these disparities, we examined the signal characteristics of the subjects, focusing on Event-Related Desynchronization/Synchronization (ERD/ERS) phenomena [42,43].

Utilizing Pfurtscheller’s method [44], this study plotted the ERD/ERS characteristic curves for subjects A03, K3b, A02, and A06. The analysis employed Eq. 17, where A is the signal energy during motor imagery and R is the baseline signal energy, measured during a reference period set from -3 to -2.5 seconds. This detailed approach underscores the significance of signal characteristics in enhancing classification accuracy and highlights potential avenues for improving model performance in subjects with initially low classification accuracies.

$$ERD/ERS={\displaystyle \frac{A-R}{R}}\times 100\%.$$

(17)

Figure 5a and Figure 5b illustrate the ERD/ERS characteristic curves for subjects A03 and K3b, respectively. During motor imagery, movements of the left hand, right hand, both feet, and tongue generally exhibit distinct ERD/ERS characteristics, resulting in better classification outcomes. Figure 5c presents the ERD/ERS characteristic curves for subject A02, where consistent trends and similar energy changes are observed in the C3, C4, and Cz channels during different imagined movements. Figure 5d displays the ERD/ERS characteristic curves for subject A06, where significant ERD is observed in the Cz channel across all fthe proposed method imagined tasks, with consistent trend patterns. These results indicate that subjects A02 and A06 did not elicit corresponding responses in the brain regions associated with different limb movements, lacking separability and resulting in poor classification outcomes.

Table 3 presents the decoding outcomes results of our proposed method and seven baseline methods on the two datasets. The proposed method achieved 80.38% accuracy and 0.737 Kappa on BCI-IV Dataset 2a, and 87.49% accuracy and 0.833 Kappa on BCI-III Dataset 3a, outperforming CSP [9], FBCSP [10], Deep ConvNet [19], Shallow ConvNet [19], and EEGNet. Furthermore, the proposed method achieved the highest results on both datasets compared to the recently proposed EEG-Conformer [45] and LightConvNet [39] models. This superior performance is attributed to the proposed method’s consideration of synchronized activities and dynamic signal changes during the MI-tasks, fully exploiting channel correlation and temporal dependence of the MI-EEG, resulting in better decoding outcomes.

This superior performance is attributed to the proposed method‘s consideration of synchronized activities and dynamic signal changes during the motor imagery, fully exploiting channel correlation and temporal dependence of the MI-EEG, resulting in better decoding outcomes.

3.3. Ablation Experiments

The proposed dual-branch network extracts spatio-spectral and temporal-spectral features of MI-EEG using Adaptive GCN and Bi-GRU, respectively, and performs feature fusion. We conduct comparative experiments to validate the effectiveness of multi-domain feature fusion.

Branch1:Only Adaptive GCN and CBAM are used to extract spatio-spectral features.

Branch2:Temporal-spectral features are extracted using Bi-GRU and MHA.

Our method:Combines temporal-spectral and spatial-spectral features using a dual-branch network.

The experimental results after the feature fusion of the two datasets are shown in Table 4. We present the precision and recall for each type of task to evaluate the model’s performance in recognizing various tasks. For the BCI-IV Dataset 2a, the precision and recall of each task are consistent across the three experiments. However, the classification performance of Branch1 and Branch2 is significantly lower than that of the feature fusion (Our method). For the BCI-III Dataset 3a, when only using Branch1 and Branch2 for classification, each task’s precision and recall distribution are uneven. For example, in Branch1, the left-hand task achieves a high precision but a low recall, indicating that the left-hand task is misclassified as other categories when only spatial-spectral features are extracted. In contrast, the classification results of each task after feature fusion (Our method) are more balanced and significantly higher than those of Branch1 and Branch2. This indicates that the model’s generalization performance is better after feature fusion, proving the importance of feature fusion and the complementarity of different dimensional information.

From Table 4, it can also be seen that the spatial-spectral features of the two datasets are more distinguishable than the temporal-spectral features. This is because the neural activity caused by limb movements has a distinct somatotopic organization. For example, hand movements mainly manifest in the contralateral brain hand functional area, and foot movements activate the central brain area. Compared to time-dimensional information, the spatial distribution differences of signals generated by different limb movements are more significant.

3.4. Comparison Experiments of Adjacency Matrices

The adaptive adjacency matrix in this study comprises three matrices: A, B, and C, capturing synchronous brain activity and dynamic connections. Three experiments were conducted to verify their validity:

Experiment1:Constructing the adjacency matrix using the spatial positioning information of EEG channels [46].

Experiment 2: Constructing the adjacency matrix using only matrix A.

Experiment 3:Constructing the adaptive adjacency matrix using Eq. 4 (our method).The experimental results of classification using different adjacency matrices for the two datasets are shown in Table 4. Comparing Experiment 1 and Experiment 2, adopting matrix A yielded better results than constructing the adjacency matrix based on spatial location information. This is because adjacency relationships can only aggregate information between channels in local brain regions, while synchronous brain activities occur in the whole brain region. Adjacency matrix A can establish connections across all EEG channels, making better use of channel information.

In Experiment 3, for the BCI-IV Dataset 2a, using the adaptive adjacency matrix(Our method) improves the precision of each task by 6.99%, 5.49%, 7.12%, and 5.76%, and the recall by 7.58%, 6.13%, 7.82%, and 5.59% compared to Experiment 2. For the BCI-III Dataset 3a, using the adaptive adjacency matrix(Our method) improves the precision of each task by 11.71%, 5.47%, 6.81%, and 1.33%, and the recall by 4.63%, 5.55%, 11.29%, and 13.52% compared to Experiment 2. The improvement is attributed to our method’s ability to establish whole-brain connectivity and adaptively capture the brain’s dynamic processes, thereby fully exploring brain functional activities.

3.5. Effect of Self-Connection Coefficients

Introducing the self-connection coefficient $\alpha$ aims to enhance the utilization of self-channel information, with its value ranging from 0 to 1. To verify the impact of the self-connection coefficient on classification results, we tested values at intervals of 0.1. As shown in Figure 6, classification accuracy increases with the self-connection coefficient for most subjects, reaching a peak before decreasing. This indicates that placing too much or too little emphasis on the node’s information can negatively affect classification performance. The optimal self-connection coefficient varies among subjects, with values of 0.3, 0.1, 0.2, 0.7, 0.5, 0.1, 0.6, 0.6, 0.6, 0.2, 0.4, and 0.2, reflecting the individual differences among the subjects.

4. Discussion

We propose a dual-branch Ml-EEG decoding method using adaptive GCN and Bi-GRU, which can capture the channel correlations and temporal.dependencies among multiple frequency bands.To better understand the advanced nature of the proposed method in capturing brain functional activities and temporal dependencies, we conduct a visual analysis of the adjacency matrix and the output characteristics of each stage of the model.

4.1. Channel Correlation Visual Analysis

The weight edges of the adjacency matrix A were visualized to evaluate the impact of graph convolution on MI-EEG channel correlation. Figure 7 shows the brain functional connectivity patterns of different subjects. The outer circle represents different EEG channels, and the inner lines indicate their connectivity properties. Darker lines show a greater correlation, based on Pearson correlation coefficients in adjacency matrix A. Weights are assigned to channel pairs based on correlation, selectively aggregating synchronous activity information. Connection strength varies among subjects, indicating individual differences.

4.2. Time Dependence Visual Analysis

To observe the impact of temporal dependencies on features, the features $\mathit{x}$ before capturing temporal dependencies and features ${\mathit{f}}_{tf}$ after capturing them for all samples of each subject are obtained, and the feature mean is calculated along the sample dimensions for visualization. The visualization for subjects A03 and K3b is shown in Figure 8. The horizontal axis displays sequences from various time windows, each aggregating data from 50 sample points (0.2 second time window), and the vertical axis represents the sequence features.

Before temporal dependency capture, the feature map is discrete, indicating weak connections between time window sequences. After mining, dependencies strengthen significantly. For example, feature channels 3 and 12 of subject A03 are extremely similar in color throughout the entire time axis, suggesting that these time segments aggregate each other’s information. This shows that Bi-GRU and MHA effectively extract MI-EEG temporal-dependence.

In the feature map before temporal-dependency capture, the first 25 sequences show obvious differences and prominent colors, while other sequences have small feature values. This is because the first 25 sequences correspond to the initial 125 MI-EEG sampling points (0∼0.5s period of the acquisition paradigm), which is the task cueing stage when subjects are in a preparation state. The data from this period generally lack useful information. After temporal-dependent mining, these features are no longer prominent, indicating that Bi-GRU and MHA effectively reduce irrelevance information.

4.3. Feature Fusion Visual Analysis

To visually demonstrate the classification effect of feature fusion, the t-SNE [47] method was used to embed the temporal-spectral features, spatial-spectral features, and fusion features (our method) into two-dimensional scatter plots to observe the degree of feature separation. Figure 9a shows the visualization results of subject A07. For temporal-spectral features, different tasks overlap significantly. For spatial-spectral features, the separability between left and right-hand tasks and between feet and tongue tasks is poor. The inter-class distances between tasks are increased after feature fusion significantly. Figure 9b shows the visualization results of subject K3b. From the temporal-spectral and spatial-spectral features, it can be seen that the left and right-hand tasks introduce individual other samples. However, after feature fusion, the left and right-hand tasks are completely separated from other tasks. This result indicates that integrating the time, frequency, and spatial domain information of MI-EEG can enhance the separability of different tasks.

5. Conclusions

Aiming at the problems that the channel correlations and temporal dependencies among multiple frequency bands of MI-EEG are difficult to capture precisely and efficiently, and the signal decoding quality needs to be improved, we propose a multi-domain feature classification method for MI-EEG based on a dual-branch network. This method uses Adaptive GCN and Bi-GRU to mine the brain functional connectivity and temporal dependencies of MI-EEG respectively. Meanwhile, the CBAM attention mechanism and multi-head self-attention mechanism are introduced to capture the valuable information in features of each dimension, achieving an effective fusion of time-frequency-spatial domain features of MI-EEG and improving the decoding quality. The proposed method is evaluated with two datasets, BCI - IV Dataset 2a and BCI - III Dataset 3a. The results show that our method has higher performance and provides a new approach for MI-EEG decoding.