Deep Learning for MRI-based Acute and Subacute Ischaemic Stroke Lesion Segmentation—A Systematic Review, Meta-Analysis, and Pilot Evaluation of Key Results

1. Introduction

Stroke remains a leading cause of mortality and long-term disability worldwide [1], placing a substantial burden on healthcare systems and societies [2]. The majority of strokes are ischaemic [3]. They can occur in different locations and are largely heterogeneous in appearance [3]. After stroke onset, the progression of ischaemic injury continues for minutes-to-days, depending on brain region vulnerability, cellular constituents, and residual perfusion levels [4]. Surrounding the ischaemic core, or irreversibly damaged tissue, appears a region that is functionally impaired, but potentially salvageable, known as ischaemic penumbra [5]. Accurate diagnosis during acute-to-subacute stages allows for interventions (e.g., thrombolytic drugs, surgery) that may potentially salvage the penumbral area.

Magnetic resonance imaging (MRI) technology has not only enabled the non-invasive investigation of human brain features, but also of ischaemic injuries, thanks to the high dimensionality and particularly low signal-to-noise ratio found in MR images. Segmentation of the infarcted regions in these images, as well as the normal tissues, has been important to advance stroke research and, ultimately, patient outcome. Since manual segmentation methods are time-consuming and subject to inter-rater variability, there has been a growing interest, since 2015 [6], in applying deep learning (DL) techniques to automate stroke lesion segmentation tasks and enhance their accuracy. DL methods can automatically extract intricate spatial and textural features within MR images, while requiring low-to-moderate subject matter expertise. DL also addresses long-dated machine learning-related challenges, such as discerning patterns in high-dimensional data, such as imaging data.

Not surprisingly, several methods have been proposed to automatically assess ischaemic lesions from MRI using DL. These have been analysed previously (Figure 1), but the data that pertains to segmentation of ischaemic stroke lesions have not been systematically reviewed, neither have been meta-analysed, nor their outcomes have been independently evaluated. We systematically review the literature from 2015 to 2023 to investigate the accuracy and generalisability of the proposed DL methods in acute-to-subacute stroke lesion segmentation on MRI, focusing on details of DL architectures and attention mechanisms, seeking to answer the following question: What would be the optimal deep learning model architecture for acute and subacute ischaemic stroke lesion segmentation on brain MRI? After meta-analysing the relevant data extracted from the sources reviewed, we conducted a pilot analysis to evaluate as many of the elements identified in the review as possible.

2. Background

2.1. Deep Learning Architectures

Convolutional neural networks (CNNs) are useful architectures for processing data with grid-like topology (e.g., 2D/3D grid of pixels/voxels) [7]. They employ convolution blocks to produce “feature maps” through the use of sparse inter-layer interactions, with kernels smaller in size than the input [8]. A standard convolutional block in a CNN (Supplementary Figure S1a) consists of a linear convolution operation on a kernel, which produces a feature map that is passed through an activation function to introduce non-linearity and enable the network to learn more complex relationships in the data [9], before it gets down-sampled by a pooling operation.

CNNs are widely used in medical image segmentation [10], with an architecture that typically ends with fully-connected layer(s) responsible for doing the predictions (e.g., pixel/tissue classification). Predictions are connected to a cost or loss function which measures their discrepancy with ground-truth data. Network parameters are then optimized through backpropagation, by minimizing the loss function until convergence, often aided by regularization methods [9]. However, (i) they produce feature maps with lower spatial dimensions than the input image, and (ii) they classify individual pixels using patches extracted around each pixel, and those often overlap significantly, which in turn creates redundancy in convolution operations. Fully Convolutional Networks (FCNs) address both drawbacks (i) by replacing CNN's fully-connected layer(s) with “up-sampling convolutions” that output images of the same size as the input, and (ii) by generating likelihood maps instead of pixel-by-pixel predictions. However, the FCN’s output maps are of particularly low resolution [6].

U-Net architecture was first used for image segmentation in 2015 [11], and it has since achieved overwhelming success. It uses a symmetric encoder-decoder structure based on convolutional blocks, where down-sampling (encoder) operations compress images and up-sampling (decoder) operations restore them, until they reach the input image’s original size [12], as opposed to FCNs. U-Nets also introduce skip connections that connect encoder-decoder layers of equal depth, hence allowing them to train with limited data while avoiding the vanishing gradient problem [13].

The ResNet architecture was published shortly after U-Net [14], to further tackle the vanishing gradient problem, also using skip connections. A standard ResNet block (Supplementary Figure S1b) consists of an “identity path” (green arrow in the figure) that can bypass the “residual path”, thus giving the network the option to simply copy activations to the next layer and preserve information when learned features do not require more depth. Skip connections also tackle the degradation issue, where adding layers leads to higher training error since accuracy gets “saturated” as the network keeps learning the data [15]. ResNets can improve model convergence speed [16], but since most residual blocks only slightly change the input signal, they produce a large amount of redundant features [17]. This is where DenseNets help.

The first DenseNet architecture was published shortly after ResNet [18]. It employs dense connections interconnecting all layers in order to maximize information and gradient propagation [13]. A standard Dense block is represented in Supplementary Figure S1c. Original inputs and activations from previous layers are both kept at each block, hence preserving the global state, while encouraging feature reuse with less network parameters [12]. Reusing features across layers also allows DenseNets to tackle the vanishing gradient problem [19].

2.2. Attention Mechanisms

When our eyes focus on a certain object, groups of filters within our visual perception system are used to create a blurring effect so that the object of interest is in focus, and the rest is blurred [20]. Attention mechanisms attempt to achieve the same “blurring effect” but for machine-based image processing. Attention can capture the large receptive field and retrieve underlying contextual details by modelling the relationships between local and global features [21]. In this work, we categorize attention mechanisms as “spatial”, “channel”, or “hybrid”.

“Spatial attention” (Supplementary Figure S2a) is responsible for generating masks that enhance the features that define a specified object (e.g., lesion) on a given feature map, therefore enhancing the input to subsequent layers of a network [20]. Examples of spatial attention methods include attention gates, i.e., computational blocks to implement “attention” as described above; self-attention, which operates solely on input sequences, thus enabling a model to further exploit spatial relationships within input scans [22]; and cross-attention (e.g., Gomez et al. [23]), which enables the network to simultaneously process encoder and decoder features, in order to pass the most aligned encoder features with respect to decoder features of same depth, and therefore decrease noisy signals in skip connections [22].

“Channel attention” (Supplementary Figure S2b) refers to the process of assigning a weight to each feature map or channel, emphasizing those that contribute most significantly to the learning [20]. Conversely, spatial attention assigns weights to pixels. Each map specializes in detecting specific features (e.g., horizontal edges, brain anatomy). Examples of channel attention methods include squeeze-and-excitation blocks [24], which were used by Woo et al. [25] and Lee A. et al. [26]).

In summary, channel attention focuses on the importance of different feature maps, while spatial attention focuses on the importance of specific regions within a feature map.

“Hybrid attention” combines spatial and channel attention. Examples include dual attention gates, which combine spatial and channel attention gates (sAG+cAG) [27]; and multi-head attention, which uses parallel processing by applying attention across multiple "heads" simultaneously, where each head may be configured to implement any channel or spatial attention operation [22].

3. Materials & Methods

3.1. Protocol Registration

We registered this systematic review protocol with the International Prospective Register of Systematic Reviews (PROSPERO), registration number: CRD42023481551 (November 2023). We conducted our review following the PRISMA guidelines [28,29].

3.2. Search Strategy

We conducted a literature search (January 2015 – December 2023) for papers published in IEEE Explore, MEDLINE, ScienceDirect, Web of Science, PubMed, Springer, and OpenReview.net. We identified keywords by expanding five subject components: accuracy, acute ischaemic stroke, deep learning, lesion segmentation, and MRI.

We also did citation tracking of reviewed articles, and hand-searching of the two journals “Stroke” and “NeuroImage: Clinical” (Recall:100%). Two reviewers (M.B. and M.C.V.H.) conducted the main search, paper selection, and data extraction, and discrepancies were resolved by discussion. The full search strategy is provided in Supplementary Data A.

3.3. Eligibility Criteria

Table 1 summarizes the selection criteria, justifying the basis for inclusion and exclusion of the different articles found during the search. Briefly, studies were included if presented (a) deep-learning algorithm(s)/architecture(s) for segmenting ischaemic stroke lesions in acute and subacute phases in humans, from MRI, and were peer-reviewed and indexed in any of the databases searched. Studies were excluded otherwise.

Studies with sufficient information to be reproduced.

Studies not reporting segmentation performance scores.

3.4. Data Extraction

For each paper, we extracted the following information: primary outcomes and measures, image acquisition protocol(s), sample characteristics, ground-truth data, data pre-processing, learning approach, model architecture, model training, model hyper-parameters, model validation, external validation, performance results, and generalisability of the proposed approach as per custom calculation. To cross-check data entry, a reviewer (M.C.V.H.) performed double extraction independently and blind to prior extraction results.

3.5. Data Analysis

We analysed the extracted results using custom-built scripts in python. We calculated fixed-effects and random-effects as part of a whole group analysis. For these analyses we used the reported dice similarity coefficients (DSC) and their 95% confidence intervals (CI) to estimate the effect size. For the effect estimates we used the weighted average of the reported mean DSC. We further divided the studies in two groups: (i) studies using attention mechanisms, and (ii) studies not using attention mechanisms and repeated the analyses for each group. We also conducted a sensitivity analysis using the precision metrics (instead of the DSC) to estimate the effect size. Lastly, we conducted a meta-regression analysis to assess whether there is statistically significant relationship between the presence of attention mechanisms and the likelihood of high mean DSC across studies. We further used the DSC and the standard errors for generating a funnel plot, followed by the Egger’s test, to assess possible bias in the meta-analysis.

3.6. Publication Quality Analysis

We assessed the sources selected following the NIH’s Study Quality Assessment Tool (https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools).

3.7. Pilot Analysis

We conducted a pilot analysis leveraging the findings from our literature analysis in an independent and publicly available sample. The specific aims of this pilot were two-fold:

• Proposing an architecture that leverages the findings of our systematic review in terms of best development practices:

  ○

  Use 2D model with image-wise training

  ○

  Increase network depth while leveraging the power of skip connections by combining U-Net and ResNet

• Conducting multiple experiments (24 in total) assessing segmentation performance in different scenarios:

  ○

  With versus without attention mechanisms

  ○

  Using a compound loss function versus a region-based loss function

  ○

  Using input images of a single modality (DWI) versus input images of multiple modalities

3.7.1. Dataset

We used the ISLES-2015-SISS dataset, published by the MICCAI 2015 conference [30]. It consists of brain MRI from 28 subacute stroke cases to use for model training. For each case, a set of five MRI sequences are provided: T1-weighted (T1-WI), T2-weighted (T2-WI), diffusion-weighted (DWI), and fluid-attenuated inversion recovery (FLAIR), along with the corresponding ground-truth masks. The data were already anonymized by removing patient information from files and facial bone structure from images.

3.7.2. Data Pre-Processing

The following data pre-processing steps were conducted:

• Intensity-based normalization using Min-Max scaling
• Intensity-based skull-stripping using BET2 (performed by challenge organizers)
• Rigid co-registration to the FLAIR sequences (performed by challenge organizers)

3.7.3. Segmentation Architecture, Model Training and Evaluation

We implemented the DL architecture, AG-UResNet50, inspired by multiple papers [31,32,33,34,35,36,37], especially Guerrero et al.’s UResNet [34], Jin et al.’s RA-UNet [36], and Gheibi et al.’s CNN-Res [37]. AG-UResNet50 is a five-level end-to-end U-Net (Figure 2), with a ResNet50 replacing its encoder path [38]. Using U-Net in combination with ResNet50 allows us to leverage the power of skip connections further [39], and make the network deeper. This makes it easier for the gradient to flow from output layers back to input during back-propagation, while handling the vanishing gradient problem. Zhang et al. [40] identified ResNet as an architecture that can improve segmentation of small lesions. Max-pooling was used for down-sampling the first set of feature maps produced by the model, since it can extract extreme features (e.g., lesion edges) well. Convolution blocks with stride two were used for remaining down-sampling operations, in order to better retain image details [13]. On the decoder side, we simply used the U-Net’s deconvolution blocks, but with Leaky ReLU activation instead of ReLU, in view of its better results in medical image analysis [41], as also demonstrated by Karthik et al. [42]. We kept the up-sampling interpolation algorithm, which basically inserts new elements between pixels in the image matrix. Feature maps from the encoder are combined with those from the decoder in the same depth using concatenation. “Attention concatenation”, which was used here, works by incorporating attention gates (AGs) in skip connections [43], as seen in Karthik R. et al. [44], Nazari-Farsani et al. [45], and Yu et al. [46]. An AG takes two input vectors that are added element-wise (Figure 3), resulting in aligned weights becoming larger and unaligned weights smaller. The output vector then goes through ReLU activation, 1x1 convolution, and sigmoid activation to produce the attention coefficients/weights. Coefficients are then up-sampled to the original dimensions of the input vector using trilinear interpolation, before being multiplied element-wise. The final output is passed along in the skip connection.

During training, we used a compound loss function mixing Binary Cross-Entropy (BCE) and Dice loss. BCE loss computed the gradient based on the difference in probability distribution of each pixel in the predicted versus real sample [47], while Dice loss directly computed the gradient using the Dice score of predicted versus real samples [18]. From a regularization standpoint, we used pixel dropout, learning rate adjustment and data augmentation methods, while for optimization, we used Adam function and batch normalization. From a training infrastructure standpoint, the model was developed, trained and tested on Azure Databricks (python:Torch), using one sizeable driver: CPU:16 cores; OS:Ubuntu; RAM:56GB; Runtime:13.2ML. We evaluated the model performance using DSC, and used five-fold cross-validation. The full code used for this pilot is available from GitHub (https://github.com/Elpazzu/UoE-Pilot-Analysis/)

4. Results

4.1. Search Results

The search yielded 1485 papers, of which 39 were ultimately retained (Figure 4).

All papers had segmentation as primary outcome. Fewer had prognosis (6 studies) or functional (3 studies) outcomes. Prognosis studies were either trying to predict tissue fate or lesion volume (e.g., Wong et al. [48], Wei et al. [49]). Functional studies mostly tried to predict the modified ranking scale score (mRS). Only one paper explicitly had diagnosis as primary outcome, but practically, segmentation and diagnosis are tightly linked, since by segmenting lesion pixels, the algorithm is effectively helping physicians with the diagnosis.

4.2. Sample Characteristics

As Table 2 shows, patients were all adults of 18 years old and above, and males were generally slightly over-represented (58% on average), except in few studies where the opposite was true (e.g., Moon et al. [50]). From a stroke severity standpoint, reported mean NIHSS [51] were always on the “minor” or “moderate” ranges (7 studies). Although both subacute and acute stroke stages were in scope, most studies (23/39) included exclusively acute ischaemic stroke cases. Reported patient mean “time-since-stroke” (TSS) were also exclusively in the acute interval, with 2 studies actually very close to the hyperacute-acute limit. Only four papers used sample sizes above 500 (Mean 252.2), and samples were most often collected from only one center (28 studies versus 14 leveraging multiple centers). Supplementary Figure S3 shows a graphical illustration of the sample characteristics.

4.3. Imaging Acquisition and Manipulation

Table 3 shows the imaging data extracted from the reviewed sources, and Figure 6 plots the correspondence between the dimensions of the images used as input to the reviewed algorithms (i.e., 2D, 2.5D, or 3D) and the spatial resolution and the manipulation of these images during training (i.e., patch-wise or image-wise). Most studies (25/39) used images of high or very high spatial resolution. DWI modality was by far the most used modality (37 studies), followed by FLAIR (17 studies). Also, two-thirds of studies adopted a multi-modal approach. Twenty-seven studies used a 2D-based approach and 10 a 3D-based approach (Table 3). 2D models exclusively used high or very high-resolution images, and 3D models exclusively moderate or low-resolution images, which seems counter intuitive (Figure 6a). 3D models adopted patch-wise training in 8/10 studies (Figure 6b). During the generation of the ground-truth, 31/39 studies reporting mismatch between image sequences; 13 studies leveraged diffusion-perfusion (DWI-PWI) mismatch, and 12 DWI-FLAIR mismatch. The magnetic field of the scanner(s) was 1.5T and 3T in 25 studies, only 3T in nine studies, and only 1.5T in three studies. See pie charts in Supplementary Data B.

4.4. Data Pre-Processing

Eighteen studies used proprietary datasets (Table 4), 24 used one or a combination of ISLES-2015 [30], ISLES-2017 [52] or ISLES-2022 [53], and three used data related to the DEFUSE or iCAS studies [54,55,56]. In relation to skull-stripping, 37 studies performed an intensity-based approach (using BET2/ITK software), one study used an atlas-based approach (Moon et al. [50] using Kirby/MMRR template), and one study used DL to reduce sensitivity-to-noise [57] (Liu C. et al. [27] using in-house "UNet BrainMask"). Inter-patient image registration onto a standard space (e.g., MINI) and/or intra-patient registration (e.g., registration of different sequences) were performed in 27 studies.

4.5. Deep Learning Architectures

Within the 37/39 studies that performed semantic segmentation, 35 studies used U-Net-based models (Figure 7). But none of them used the original U-Net as-is [11], with perhaps Cornelio et al. [58] and Aboudi et al. [59] being the closest. ResNet architecture was the second most used (8 studies), while DenseNets were only used in three studies. Data augmentation was the most used regularization method (28 studies), whereas each of dropout, early stopping, weight decay, class weighting, and learning rate adjustment were used in 10-11 studies. More papers used image-wise training (27 studies versus 14 for patch-wise training); 7/8 studies that were dealing with smaller mean lesion volumes (< 40ml) used patch-wise training. In addition, none of the papers performed uncertainty quantification, and 31 algorithms were end-to-end (versus 8 multi-module). Twenty-three studies used Dice loss (Table 5), either mixed with other loss functions (8 studies) or standalone (15 studies). Cross-entropy loss was used in 17 papers, nine times standalone. Focal loss was only used in four papers, and two papers used Liu et al.’s custom-built loss function [16]. Twelve studies used attention: five used hybrid attention, four spatial attention, and three used channel attention (Table 5). Studies deploying ResNet-based architectures did not incorporate attention.

4.6. Performance and Generalisability

As Table 6 shows, the most used performance metrics across the studies reviewed were the overlap metrics Dice, Recall, and Precision, as well as the Hausdorff distance [61]. Six papers only used one single metric. To comparatively evaluate the models according to their performance, we assigned a generalisability score to each of the included studies based on sample representativeness – considering sample size, number of study sites, gender equality, age range, length of the data collection period, number of scanners, external validation performed –, ground-truth data, and access to clean code (Table 6, third column from right to left). Liu C. et al.’s algorithm [27] was deemed “highly” generalisable, whereas 19 algorithms had “low” generalisability. Plotting the reported performance against the generalisability scores obtained revealed that Dice and generalisability scores were positively correlated (Supplementary Figure S4a).

Only six papers analysed segmentation performance in relation to lesion size (i.e., on small versus large lesions), and in four of them, accuracy on small lesions was lower or significantly lower (Figure 8b). As shown in Supplementary Figure S4c, lesion volume ranges differed substantially between studies, and all cases with low mean Dice (< 0.6) (10 studies) reported low mean lesion volumes (< 40ml), while all cases with higher lesion volumes (> 60ml) (4 studies) reported high Dice scores (> 0.68). In other words, segmentation performance was generally better when lesions were larger.

As shown in Figure 8a, Dice scores were above the overall mean and relatively consistent across T2-WI, T1-WI, and FLAIR imaging modalities (mean Dice around 0.7), while PWI exhibited lower-than-average performance (mean Dice 0.38). Only for DWI did all the data points fall within the IQR (between 25^th-75^th percentiles), as outliers with below-average Dice scores were observed for FLAIR (three), T1WI (two) and T1WI (one). Additionally, the lower half of the IQR (25^th-to-50^th percentile) was substantially wider than the upper half (50^th-to-75^th percentile) for DWI, whereas the opposite pattern appeared in the IQR for PWI.

We also saw a positive correlation between spatial resolution and reported segmentation performance (Supplementary Figure S4b). Seven studies performed external validation of their models on unseen data. Most of them obtained higher Dice values on the study’s test set than on the external validation set. We also observed a positive correlation between sample size and segmentation performance. Also, single-centre studies showed better performance (mean Dice 0.71) than multi-centre studies (mean Dice 0.6).

Dice scores were much higher for studies using ISLES-2015 (mean Dice > 0.7) or proprietary datasets (mean Dice 0.72), than when using ISLES-2017 (mean Dice 0.38) or DEFUSE (mean Dice 0.52). When attention-based networks were deeper, or when U-Nets were deeper, Dice scores were higher (Supplementary Figure S4d). The mean Dice was also higher when attention was used (0.71 versus 0.6 if not used).

Models using focal loss heavily under-performed, while those using learning rate adjustment over-performed. There was negative correlation between Dice scores and numbers of epochs used. Interestingly, only one of the algorithms that used a relatively high number of epochs was also using early stopping regularization, which means that for all the others, the full (high) amount of epochs was used during training, hence substantially increasing the probability of overfitting.

4.7. Reported Dice Scores and Segmentation Quality

We explored whether the reported Dice scores are a legitimate indicator of segmentation quality in this review. For this we generated a forest plot using the data from the 17 papers that reported their Dice along with their standard deviation (Figure 9). In this analysis, the percentage of variation across studies due to heterogeneity rather than chance (I²) was 22.08%.

We also conducted a sensitivity analysis using Precision scores as effects size instead of Dice scores. This analysis involved only eight studies, which reported their precision scores along with standard deviations. But in this analysis, I² was 8.49%, indicating a reduced level of heterogeneity between studies, therefore precluding us to derive conclusions from it (Supplementary Figure S5).

Funnel plots and Egger’s tests (Supplementary Figures S6 and S7) conducted using the Dice scores reported by the included studies indicated the presence of publication bias in favour of studies reporting high values of this metric.

4.8. Influence of Attention on Dice Scores

We conducted a subgroup analysis to evaluate the association between attention mechanisms and Dice scores. The resulting forest plot is shown in Figure 10.

There were no statistically significant differences in effect sizes between the groups. The subgroup “with attention” indicated moderate heterogeneity in I² (31.63%) and a very high Z-stat (39.03, p<0.001), suggesting a substantially large overall effect. While this implies that the presence of attention may enhance segmentation performance, the small number of studies in this subgroup (five) limits the conclusiveness of this result. In contrast, the subgroup “without attention” comprised 12 studies, showing significant heterogeneity in the Q-stat (Q=19.93, p=0.05) and in I² (39.80%). Despite the absence of attention, a significant overall effect was also observed (z=5.27, p<0.001). This suggests that when attention is not used, the Dice scores differ between studies.

Further meta-regression analysis to assess the statistical significance of the relationship between “attention mechanisms” and “Dice scores” (Supplementary Figure S8) revealed that 8.1% of the variance in Dice scores was explained by the presence of attention (R-squared: 0.081), but the slope indicating the change in Dice associated with the presence of attention was not statistically significant (0.117, p=0.27, 95% CI of the slope [-0.100,0.334]). This indicates that from the literature analysis we cannot conclude that the presence of attention has a significant impact on the likelihood of high Dice.

4.9. Risk of Bias Assessment

After assessing the possibility of biases in the included studies, 31 studies scored “GOOD”, and eight scored “FAIR” in the NIH study QA (Supplementary Data C). Although these results are positive, we identified cases of potential spectrum bias [107], mostly due to the following factors: acute stroke studies were more represented than subacute (30 versus 16), exposure was often only assessed once (i.e., no follow-up scans) (24 studies), variance and effect estimates were not both provided (21 studies), few experiments were conducted to assess the different levels of exposure related to the outcome (11 studies), period of data collection was relatively short (10 studies), study population was poorly defined (3 studies), and the age range of participants was not always consistent (e.g., Kim et al. [63] only included patients between 58-79 years old).

We also noticed cases of selection bias. Multiple studies used the same ISLES datasets to evaluate the performance of their segmentation methods. This, although advantageous (e.g., cost effective, allows comparability), introduces selection bias. These were also studies where males were over-represented in the sample.

Also, ground-truth data were most often obtained by manually refining semi-automatic segmentations (e.g., thresholding followed by region-growing), which introduces observer bias. Sixteen studies did not provide information about labelling criteria, so it is unclear whether observer bias was present in those.

We identified two other forms of bias: verification bias in 10 studies, where only one expert did the labelling of ground-truth images, and measurement bias, as mean Dice scores on ISLES-2017 were generally much lower than those on ISLES-2015, and when segmentation performance was reported for small versus large lesions, the definition of a small and a large lesion (in ml) was not consistent across studies.

4.10. Pilot Analysis

The best performing model was “UResNet50” on DWI (single-modality approach), using a weighted compound loss (BCE = 0.3 + Dice = 0.7), with a Dice score on the validation set of 0.692 ± 0.132 (Table 7).

The second best was “AG-UResNet50” (0.676 ± 0.222), with a single-modality approach, and using the same compound loss (BCE = 0.3 + Dice = 0.7).

Experiments with “UNet” and “AG-UNet” generated relatively poor Dice scores. Performance was better in single-modality experiments. Abdmouleh et al. [64] made the same test on the same dataset, but they achieved quasi-equal performance in their DWI-only and multi-modal experiments (Dice 0.71). Performance was also better when using compound loss “BCE = 0.3 + Dice = 0.7” versus the other two types. The 12 experiments using attention and the 12 not using attention yielded similar average Dice scores.

Average training times for UResNet50 was 5h 43min, for U-Net it was 5h 31min, for AG-UResNet50 it was 6h 15min, and for AG-UNet it was 5h 55min. Multi-modal experiments took longer to train in all cases (~3 hours longer each time). Same was true for attention-based experiments (~30 minutes longer each time).

5. Discussion

5.1. Systematic Review and Meta-analysis

Although our review protocol did not have age restriction, samples never included patients below 18 years old. This stresses the lack of research in paediatric stroke, which may be due to multiple factors, e.g., delayed identification of stroke, numerous stroke aetiologies and risk factors in children, and limited imaging data‌ [65].

The underrepresentation of females in studies, on the other hand, can be partially explained by the difficulty of diagnosing females with stroke, due to factors such as higher proportion of stroke mimics (e.g., migraine), pre-stroke disability, or neglect of symptoms among females [66]. These uneven distributions of gender and age data can affect the universality of our research outcomes.

Most studies focused exclusively on minor-to-moderate stroke cases with focus on acute stroke, since DWI and FLAIR are able to show high signal in AIS-affected brain areas, whereas signal begins to diminish gradually towards the subacute stage, often leading to lower MRI sensitivity for stroke identification [53]. Such differences in MRI signal between subacute and acute lesions give the idea that combining acute and subacute cases in one single dataset, as seen in Liu C. et al. [27] and Liu Z. et al. [67], might require highly trained observers to manually delineate the lesions (i.e., generate the reference labels).

We also noticed relatively small sample sizes across studies, which is not new in AIS research [68]. Data augmentation is a common way to mitigate this issue, and Clèrigues et al. [69] proposed a novel “symmetric modality augmentation” technique, which leveraged learned features based on the symmetry of brain hemispheres. Other ways to deal with small sample sizes include active learning (e.g., Olivier et al. [70]), semi-supervised learning using weakly labelled data (e.g., Zhao et al. [71]), or transfer learning (e.g., Li et al. [72] used TernausNet [73] which was pre-trained on ImageNet [74]).

Several studies used high spatial resolution images to capture more fine-grained features from the data and improve segmentation performance on small lesions. Other deepened their networks further to collect more nuanced features, but the higher the number of down-sampling operations, the lower the resolution of the feature maps, to a point where reconstructing lesions in the up-sampling path becomes virtually impossible. Furthermore, risks of overfitting/over-learning increase substantially when networks are deeper, especially in absence of skip connections.

More generally, using 3T magnetic field strength, as done by 34/39 studies, can also help with small lesions, as it offers better signal-to-noise ratio and spatial resolution versus 1.5T, and it reduces imaging artifacts by offering more uniform B1 inhomogeneity [75].

Most studies used DWI, known as the gold standard for early stroke detection [76], and many used T1-WI, a staple in subacute stroke research [77], T2-WI, PWI, or FLAIR. PWI was frequently applied to detect the ischaemic penumbra [57], and most used FLAIR as it offers enhanced lesion clarity by suppressing CSF details [78]. For instance, Khezrpour et al.‘s U-Net used only FLAIR and got very high accuracy [79]. ADC maps were also often used with DWI for more robust ground-truth data, as lesions appear simultaneously hyperintense on DWI and hypointense on ADC in early stroke stages.

The impact of using different imaging modalities (i.e., T1-WI, T2-WI, DWI, PWI, FLAIR) on lesion segmentation accuracy was also observed, as each modality may highlight distinct pathological features, which may, in turn, influence algorithm performance.

DWI-PWI mismatch [80] was commonly used to create ground-truth sets (e.g., Lee S. et al. [81]), since PWI identifies penumbral tissue, while DWI delineates the core infarct (i.e., areas of restricted water diffusion [77]). Despite its utility though, DWI-PWI mismatch analysis remains challenging. Establishing clear imaging boundaries for recoverable tissue is not straight-forward [77]. Large perfusion abnormalities may be observed in patients without corresponding clinical deficits [82]. There is no universally defined mismatch ratio, although Kakuda et al. tried to define one [83] DWI-FLAIR mismatch, on the other hand, is mostly used for TSS assessment in hyper-acute-to-early-acute stage [84]. Combining both mismatch analyses can definitely help experts effectively delineate stroke lesions.

Many argue that using 3D images is crucial for DL-based stroke lesion segmentation, but (i) few methods address the associated computational challenges [85], which explains why the majority of retained studies used 2D images, and (ii) 3D models proved to be good at segmenting large organs, but are less “established” in stroke lesion segmentation [10].

Cutting 3D images into 3D patches (i.e., patch-wise training) is a way to mitigate both the computational challenges, by reducing memory overhead [13], and the small lesions challenge, by forcing the model to focus on a smaller area of the entire image. That explains why eight out of ten 3D studies in this review have used patch-wise training.

On the other hand, the majority of studies that used ISLES-2015/2017 have processed those as 2D images, mainly due to their low-resolution when processed as 3D (slice thickness: 5mm). However, it was surprising to see so many 3D models use low resolution images, since the whole point of 3D models is to capture detailed information from images [86]. For instance, Zhang R. et al. [19] proposed a 3D model that captured both low-level local features and high-level ones, but they used low-resolution images.

Dice was the most used performance metric across studies, as (i) it is simple to interpret, (ii) it handles class imbalance, and (iii) its widespread use facilitates comparison between different methods. However, it remains an overlap metric that is prone to instability, especially with small lesions [87], and for an evaluation to be holistic, it must be accompanied by other types of metrics (e.g., surface-based, boundary-based, volume-based).

Dice scores were higher for single-center studies, but since too few of these studies performed external validation, we cannot exclude “over-adaptation” to the image acquisition protocol(s) from that one center, and therefore poor model generalisability.

Moving on to loss functions, CE loss quantifies the difference between two probability distributions (e.g., predictions and ground-truth), but it cannot handle class imbalance since each pixel/voxel contributes equally to the loss, and therefore the learning process may easily fall into a local optimal solution [88].

Focal loss is an adaptation of CE loss that introduces a modulating factor aimed at down-weighting the impact of well-classified examples [89], but since “lesion” is already the minority class in our case, focal loss overly penalizes correctly classified lesion pixels, which explains the very bad performance of studies using it (e.g., Hu et al.’s Brain SegNet [90]).

Generally, overlap-based loss functions (e.g., Dice loss) are more robust to data imbalance issues [89]. By penalizing false positives and false negatives differently, Dice loss indirectly encourages better performance on minority classes. However, despite its common usage, Dice loss has some limitations [89]: it fails to capture the distance between non-overlapping but close lesions, overlooks precise contour details (combining it with a boundary-based loss may help), and it disproportionately penalizes small lesions, especially in presence of large lesions, as opposed to distribution-based loss functions (e.g., CE loss) which have no such bias. A few custom loss functions have also been proposed to address class imbalance (e.g., Rachmadi et al.’s “ICI loss” [91], loss with data fusion [92]).

Since most studies were U-Net-based, they primarily performed semantic lesion segmentation. Perhaps the fact that only two studies did instance segmentation is linked to the difficulty of delineating individual lesions in presence of motion artefacts and irregular shapes [93,94], as shown by Wu et al. [87].

Meanwhile, several studies proposed quite innovative methods:

• Liu Z. et al. [67] proposed a ResNet and a global convolution network-based (GCN) encoder-decoder. Each modality was concatenated to a three-channel image, then passed as input image to a series of residual blocks. The output of each block was then passed to its corresponding up-sampling layer using a skip connection incorporating a GCN and a boundary refinement layer
• Liu L. et al.‘s “MK-DCNN” [95] consisted of two sub-DenseNets with different convolution kernels, aiming to extract more image features than with a single kernel by combining low and high resolution
• Three studies proposed “ensemble mechanisms” (i.e., different networks that process the same input in parallel) in order to reduce overfitting, since sub-networks can learn different features from the data [13] and/or to decrease prediction variance (e.g., Choi et al. [96])
• Wu et al.’s W-Net [87] tackled variability in lesion shape by trying to capture both local and global features in input scans. A U-Net first captures local features, which then go through a Boundary Deformation Module, then finally through a Boundary Constraint Module that uses dilated convolution to ensure pixels neglected in previous layers can also contribute to the final segmentation
• Pinto at al. [97], Duan et al. [98] and Zhang L. et al. [99] proposed “information fusion mechanisms” that effectively fuse different features either from multiple modalities, or multiple plane views, thus improving their models’ ability to capture intricate lesion features
• Lucas et al. [100] added to their U-Net skip connections around each convolution block, besides those linking encoder-decoder layers

The main purpose of attention mechanisms is to address the loss of information during down-sampling and up-sampling operations. Self-attention was often used across studies, since it allows the model to capture global dependencies within the input data, which can help in identifying subtle features that span across larger regions.

Overall, there were several interesting implementations, or pseudo-implementations, of attention:

• Karthik et al. [101] embedded multi-residual attention blocks in their U-Net, hence allowing the network to use auxiliary contextual features to strengthen gradient flow between blocks and prevent vanishing gradient issues
• Vupputuri et al. [102] used self-attention through multi-path convolution, aiming to compensate for information loss, while using weighted average across filters to provide more optimal attention-enabled feature maps
• Ou. et al. [103] used lambda layers, which work by transforming intra-slice and inter-slice context around a pixel into linear functions (or "lambdas"), which are then applied to the pixel to produce enhanced features. As opposed to attention, lambdas do not give “weights” to pixels

I believe that it is only a coincidence that ResNet-based models never incorporated attention across reviewed studies, as numerous relevant publications combine ResNet with attention [104,105,106].

In terms of optimization methods, RMSProp can be effective in DL (e.g., Ou et al. [103]), as it is able to discard history from the extreme past and thus enable rapid convergence during training. However, Adam remains the most popular method as it incorporates momentum, which speeds up the optimization of model parameters, while performing bias corrections to improve the accuracy of gradient estimates during training. Also, Adam's default hyperparameters often work well in DL, mainly thanks to the adaptive learning rates which allow smooth parameter updates even in presence of noisy gradients.

While never performed, uncertainty quantification to obtain true network uncertainty estimates [107] is of utmost importance to promote the use of such algorithms in clinical practice, as it would allow physicians to assess when the network is giving unreliable predictions [6].

The generalisability of our studies was generally low, for issues that have already been highlighted above (e.g., small sample sizes, loose verification of labelled data), but researchers can easily improve the generalisability of their models by performing external validation, publishing their code, combining image acquisition protocols, and/or combining data from multiple centers.

Our risk of bias assessment yielded fairly good results. However, several instances of potential or actual bias warrant attention:

• Findings drawn from reported performance metrics (e.g., Dice) must be carefully interpreted, as performance depends on the quality of the data being used, which was variable across studies
• Results of this review may be skewed towards acute stroke (rather than subacute)
• Over-reliance on specific public datasets, which may have selection biases, may limit the generalisability of the research findings, as reported results may not fully represent all possible clinical scenarios
• Findings in terms of segmentation of small versus large lesions are slightly flawed, due to the various ways in which these two categories were defined across studies
• Data augmentation helped reduce overfitting by increasing the size of the training data, but effects of bias cannot be balanced-out by increasing the sample size by repetition [62]

Our whole group analysis included 17 studies, which is enough to consider findings meaningful [108]. The random-effects model worked better for us, which is aligned with the literature, where RE is considered a more natural choice than FE in medical research [109].

The most interesting finding resulted from the subgroup analysis. It is the uncertainty in the evidence that incorporating attention into DL architecture for AIS lesion segmentation improves model performance.

Meanwhile, the significant heterogeneity observed through these analyses may be linked to several factors, such as differences in image acquisition protocols (e.g., spatial resolution, scanners), patient populations (e.g., stroke stage, severity, aetiology), network architecture (e.g., U-Net, ResNet), model hyper-parameters, and more.

Therefore, when looking into ways to improve DL-based stroke lesion segmentation algorithms, our analysis suggests that one might want to look at factors other than attention (e.g., image quality, model architecture and complexity).

5.2. Pilot Analysis

The relatively high Dice scores obtained on training sets versus validation sets are likely caused by overfitting, partly due to the small sample size, despite efforts to mitigate this with data augmentation and pixel dropout.

We used the ISLES-2015-SISS dataset for this analysis. It is worthwhile noting that it may not sufficiently capture the variability across different populations and lesion types, and the limited sample diversity could limit the generalizability of the model across different demographics or lesion types. However, from the 39 publications analyzed, only 12 used this sample in the development of their proposed algorithm sometimes as part of a wider sample (5/12 publications) (Table 4). In terms of number of 3D volumes the sample is small, but we use a 2D model for which the number of input samples with image information multiplies the available data sources by a factor of approximately 100 considering only one dimension (e.g., considering horizontal-only or sagittal-only or coronal-only slices), but if slices in the three main imaging axes are considered, then the increase is three times that.

Not using attention yielded slightly better than using it. In this case, with a small sample size and a relatively deep network, increasing the number of learnable parameters using attention gates might have accentuated the overfitting problem. Complementing our analysis with additional performance metrics (e.g., HD, Accuracy, Precision) could further support this observation.

The fact that the single-modality approach (DWI-based) performed better than the multi-modal approach is counter-intuitive, since combining sequences has often led to an improved segmentation performance, as shown by Liu L. et al. [16] and Liu Z. et al. [67], who did the same comparison of approaches. However, it could be that specifically in the ISLES-2015-SISS dataset, the mix of image acquisition protocols across centers has introduced noise in the data, which was not properly removed during data pre-processing [57].

Compound loss (Dice + CE) outperformed Dice loss, as it was the case with Kumar et al.’s “CSNet” [110]. Since Dice loss is not suitable for small diffuse lesions, combining distribution-based loss with region-based loss has certainly helped.

UResNet50 addresses the challenge of distinguishing stroke lesions from other pathologies, which can vary by stage. Its effectiveness confers it potential to improve diagnostic accuracy and treatment planning for stroke patients, ultimately contributing to better clinical outcomes.

6. Study Limitations

• Only articles published in (or translated to) English that were accessible via institutional login were reviewed. Accordingly, relevant papers may have been missed
• Relevant papers may have been missed as a result of incongruences between search terms and article keywords in the various databases
• Since most of the included studies were not longitudinal, this review lacks an assessment of long-term patient outcomes, which is an essential factor in validating the clinical relevance and predictive value of segmentation algorithms
• While the review outlines the impact of lesion size on segmentation performance, the pilot analysis does not specifically assess how algorithms can be optimized for lesions of varying sizes

7. Conclusions and Future Works

While we included a fair number of studies in this review, the identified generalisability issues hinder the robustness of our findings. However, we were able to (i) identify the often subtle elements and configurations that can make a DL model perform better its AIS lesion segmentation task, and to (ii) demonstrate with confidence that attention mechanisms do not necessarily improve current DL architecture is AIS semantic lesion segmentation, and that other details such as model design were much more important.

We have compared multiple model artifacts (e.g., loss functions, optimization methods), discussing their potential impacts on segmentation performance. A more formal decision tree could complement our research, helping to (i) facilitate decision-making during model development, and (ii) enhance model transparency and trustworthiness in clinical settings.

In this review, algorithms were assessed solely based on performance (using Dice coefficients). A more comprehensive evaluation of their practical value could be conducted in future work by considering factors such as processing time and resource consumption.

More generally, further well-conducted and well-reported research is needed in this field, focusing on: (i) larger datasets, potentially by leveraging consortia such as the Human Connectome Project (https://www.humanconnectome.org/) or ENIGMA (https://enigma.ini.usc.edu/), (ii) higher-quality data, such as generating structured labels from radiologist reports [111], and (iii) longitudinal data to better assess how segmentation results impact patient treatment and prognosis.

Interpretability of algorithms must also improve, as today, computer scientists focus primarily on reaching higher levels of accuracy, while clinical researchers focus on verifying associations with patient outcomes [112]. For instance, deconvolution networks and guided back-propagation can explain the inner workings of DL networks [113,114].

Also, model fine-tuning remains time-consuming. Perhaps “Neural Architecture Search” will soon be a robust solution for automatic selection and parameterization of DL models [115].

At last, following the big leap DL took with the advent of GPU, many scientists are getting prepared for the next big leap, with quantum computing. Although this review did not focus on such technological advancements, the application of quantum algorithmic principles (e.g., running quantum operations on qubits) to ML has already begun [116], and expertise is being built for when quantum hardware will be commercially available. This may increase computing speed significantly.