A Hybrid Learning Framework for Enhancing Bridge Damage Prediction

1. Introduction

Bridge infrastructure is critical to our transportation networks, and ensuring their structural integrity is primary. Deterioration and damage to bridges can lead to significant economic losses, traffic disruptions, and, in severe cases, loss of life because they are susceptible to various forms of damage, such as cracks, corrosion, and erosion, which can compromise their safety and functionality. Early detection of these damages is crucial for timely maintenance and repairs, preventing catastrophic failures, and minimizing economic losses[1,2].

In the past decade, several industries have employed Artificial Intelligence (AI) algorithms to acquire information and insights, thus enhancing their performance. The concept of AI is to provide machines with the ability to think and act like human beings. It contains supervised and unsupervised learning strategies. Artificial intelligence methodologies, including diagnosis, prediction, clustering, and classification, are used to develop models that distinguish between various data classes [3]. These models are employed to predict categorical class labels, such as "Background" or "Defects," and "Crack" or "Non-crack" in the classification of bridge damage. These techniques are employed to transform data into knowledge, enabling specialists to improve their work. This knowledge is represented as models. A model exhibiting high-performance metrics indicates that the artificial intelligence methodology effectively extracted knowledge and distinguished between various data patterns. However, the extracted knowledge may not always provide the correct predictions [4].

In the context of feature extraction based on handcrafted method, Cao et al. [5] trained CNN descriptors using Bag of Visul Words (BoVW), while Hassan et al. [6] utilized color moments as a global feature and Local Binary Patterns (LBP) as a local feature to construct BoVW. Additionally, various studies [7,8,9,10] utilize BoVW for image classification and recognition. Many methods have been used to modify BoVW; for instance, an improved BoVW (iBoVW) has been proposed to generate the features from each convolutional layer of the pre-trained CNN model [11]. Also, a BoVW model presented by Olaode and Naghdy [12], whereby deep feature learning via a stacked autoencoder was used to extract image features, and for video-based recognition, spatiotemporal scale-coded BoW (SC-BoW) was utilized. It entails splitting spatiotemporal characteristics into sub-groups according to the geographic scale from which they were taken and then encoding the acquired multi-scale information into BoW representations [13]. The SO-BoVW model proposed by Sultani and Dhannoon [14] combined the Scale Invariant Feature Transform (SIFT) and Oriented Fast and Rotated BRIEF (ORB) based on the BoVW model for image classification, where k-nearest neighbor (kNN) was utilized for image classification.

Based on our analysis of concrete damage detection models mentioned in previous literature, we have seen that some studies utilize deep learning models while others employ machine learning approaches. Some studies combine image processing techniques with either deep learning or machine learning. Bhalaji Kharthik et al. [15] employed twelve transfer learning DCNN models to categorize fractures in three publicly available datasets: CCIC, SDNET for structural cracks, and BCD for bridge fissures. The researchers applied two image enhancement techniques, LBP and contrast enhancement, to the SDNET images owing to the constrained efficacy of the transfer learning DCNN models. Furthermore, the support vector machine (SVM) was trained utilizing deep features derived from the final fully connected layer of the DCNNs. The integration of deep features with SVM resulted in an enhancement of detection accuracy across all combinations of DCNN and datasets. Yang et al. [16] proposed the utilization of the DCNN framework to classify fissures across three publicly accessible datasets: CCIC, BCD, and SDNET. The proposed methodology facilitates the transfer of three distinct forms of knowledge derived from prior scholarly achievements: sample knowledge, parameter knowledge, and model knowledge. The VGG network was augmented with additional fully connected layers, thereby establishing a novel learning framework. The researchers substantiated the credibility and efficacy of the proposed methodology. Zoubir et al. [17] employed a histogram of oriented gradients (HOG) in conjunction with uniform local binary patterns (ULBP) to derive features from a dataset comprising over 3000 images, both uncracked and cracked, which encompassed various crack patterns and representations of concrete surfaces. Nonlinear dimensionality reduction techniques were executed through kernel principal component analysis (KPCA), and three machine learning classifiers were utilized to facilitate the classification process. The findings of the experiments indicate that the classification approach predicated on the SVM model and the feature-level fusion of HOG and ULBP features subsequent to the application of KPCA yielded optimal results, as the proposed classification framework accomplished an accuracy rate of 99.26%. Zoubir with other team [18] introduced a publicly accessible benchmark annotated image dataset comprising over 6900 images depicting cracked and non-cracked concrete bridges and culverts. The dataset presented encompasses a variety of challenging surface conditions and comprises concrete cracks exhibiting diverse sizes and patterns. The authors analyzed the proposed dataset employing three cutting-edge DCNNs, namely VGG16, VGG19, and InceptionV3, utilizing a transfer learning approach. The models above were employed to differentiate between cracked and non-cracked images, achieving a maximum testing accuracy of 95.89%. The outcomes of the experiments illustrate the promising applicability of this dataset for training deep learning networks aimed at the recognition of concrete cracks in bridge structures. Xu et al. [19] created a convolutional neural network (CNN)-based model for crack detection that works completely, from the beginning to the end. It does this by using atrous convolution, the Atrous Spatial Pyramid Pooling (ASPP) module, and depthwise separable convolution. An implementation of atrous convolution makes it easier to get a wider receptive field while keeping the input data's resolution. When the ASPP module is used, the network can gather contextual data at different levels, and the depthwise separable convolution makes the computations easier. The model proposed by the authors achieved an impressive detection accuracy of 96.37% for BCD dataset without the necessity for pre-training.

In this study, we aim to build a classification model which minimize the prediction errors. At first, we built a model based on LBP-BoVW, Apriori, and MobileNet algorithms. We have achieved excellent accuracy with this model, ranging from 98.27% to 100%, and very few misclassified instances with an error rate between 1.73% and 0% for all datasets. The contributions of our work are summarized as follows:

• Reliable hybrid supervised and unsupervised learning based on LBP-BoVW features for minimizing error rate.
• Built a new algorithm for reliable learning on image datasets.
• Employ the Apriori algorithm for selecting the robust features and dimensionality reduction.

We organize the rest of the paper as follows: we provide a brief explanation of the relevant works in the subsequent section. Section 2 describes the methodology of the proposed model. Section 3 presents the experiments and results. Section 4 describes the discussion. Finally, section 4 presents the paper's conclusion.

2. Materials and Methods

The proposed model contains multiple steps: preprocessing, feature extraction based on LBP-BoVW, feature selection and dimensionality reduction based on apriori algorithm, and classification using MobileNetV3_Large algorithms. Figure 1 explains the general structure of the proposed model.

2.1. Preprocessing

Depending on the purpose, nature, quality, and use of the data, one can employ various strategies, as data preparation is a crucial step in preparing the data for model training. Preprocessing the model might also boost model inference and reduce training time. Reducing the size of the input images can significantly reduce the model's training time while still preserving its performance, especially if the images are very large. This paper employs the following preprocessing techniques:

• Image resizing works the model correctly; images must be resized to a consistent size. The large image is difficult to process effectively. Therefore, we resized the different sizes of the used datasets to 100 x 100.
• Grayscale image: a grayscale instead of color is used in this work to simplify the image's data and lower the processing requirements of algorithms.
• Data augmentation techniques are used to produce new images from existing ones to increase the size of a dataset. It enhances the model's generalization and decreases overfitting.
• The data normalization technique sets pixel intensity values to a predefined range, usually between 0 and 1.

2.2. Local Binary Pattern

The LBP is a textural descriptor described by Ojala et al. [20] and utilized for classification. To calculate a binary pattern, LBP compares the intensity of a center pixel (threshold) with the intensity of the 8 neighbor pixels around it. Figure 2 shows the LBP work. LBP may be formalized as follows for a center pixel Ic and an adjacent pixel Ii (i = 1, 2, …, p) [21]:

$$LBP={\displaystyle \sum }_{i=1}^{p}F(I_i-I_c)x{2}^{i-1}$$

(1)

$$F\left(I\right)=\left\{\begin{array}{c}1, \\ 0,\end{array}\right.\begin{array}{c}I\ge 0,\\ otherwise.\end{array}$$

(2)

2.3. LBP-BoVW

The LBP-BoVW features are extracted by training the image patches using the k-means clustering technique. Where the size of the dictionary (k) is determined, which represents the number of clusters, and the cluster centroids were used as visual words. The steps of LBP-BoVW are explained as follows:

• Determine the number of patches needed to divide each image into patches. Here we used 250 image patches.
• Computed the LBP for each patch.
• To determine the size of the dictionary (k), we used 50, 100, 200, and 300, which represent the number of K-means clusters.
• Identify each cluster's center. These centers are the visual words. The size of the visual word vector is equal to K.
• Computed the histograms for each image to create the local feature vectors.

Figure 1 explains the steps above.

2.4. Feature Selection and Dimensionality Reduction

The hidden patterns and knowledge are extracted from massive amounts of data in the form of association rules, with the assistance of association rule mining. One of these association rule mining algorithms, the Apriori algorithm [22], uses frequent item set mining to generate association rules from a transactional dataset. Additionally, machine learning uses it for classification, feature selection, and dimensionality reduction. Selecting characteristic features from raw data is a fundamental stage in data mining. There are two main objectives for feature extraction. The first is to improve the classifier's efficiency by representing each data object with a limited number of useful features, and the second is to remove features that are not representative of the data object, which will increase classification accuracy [23,24]. The main objectives of dimensionality reduction and feature selection are [25]:

• We can reduce overfitting by eliminating redundant or unnecessary features from the model, especially when working with high-dimensional data.
• Dimensionality and feature reduction can produce a simpler, more interpretable model that is easier to comprehend and explain.
• Reducing execution time by shrinking the model's size, data pruning assists in speeding up the training and inference processes.
• By eliminating noisy or unnecessary features, the ability of a model to generalize to previously unobserved data can be enhanced.
• By data pruning, the best trade-off between model size and performance is achieved, resulting in a balance between accuracy and complexity.
• Feature reduction helps to enhance model performance and lower the chance of overfitting.

We used min-support = 0.9 and min-confidence = 0.9 as a threshold for feature selection. After applying the Apriori algorithm to the features derived from LBP-BoVW, the number of features that are selected in one image differs from the number of features that are selected in another image. So, the resulting features need to be processed to obtain an equal number of features for each image. To treat these missing values, we used the imputation mean method. When data is imputed, missing values are substituted with values that make sense given the information at hand. Data can be imputed in a way that maintains the set's size and structure, prevents the removal of information that can be valuable, and lowers the predictability and bias of the estimations. Table 1 explains the number of features after Apriori feature selection and the size of LBP-BoVW features for used datasets.

2.5. MobileNet

MobileNet is a computer vision model that has been made available as open source by Google. It is specifically developed for training classifiers. The network is a CNN that uses depthwise convolutions to greatly decrease the parameter count in comparison to other networks, thereby creating a lightweight deep neural network. MobileNets are compact models designed to accommodate limited resources, such as low latency and low power consumption. They are specifically parameterized to cater to a range of use cases, making them well-suited for mobile applications and other situations with constrained resources. The efficiency of MobileNet is attained by using depthwise separable convolutions, which decrease the number of multiply-accumulates and the total size of the model. Although MobileNets exhibit superior speed and compactness compared to other prominent networks, they may sacrifice some accuracy in comparison to bigger, resource-intensive models. Nevertheless, its performance remains commendable, with just a little decline in accuracy [26]. In our work, we trained the datasets using MobileNetV3-Large [27].

3. Experimental Results

3.1. Experimental Environment

In this section, we demonstrate that the proposed model is generalizable and capable of minimizing error rate by evaluating their performance on several datasets using 10-fold cross-validation.

Our model implemented in the TensorFlow and Keras environments. We developed it in Python 3.9 using the Anaconda Navigator IDE and utilized it with Windows 10 64-bit. An i7-8550U CPU runs at 1.99 GHz and 32 GB of RAM, and the experiment's settings are as follows: the batch size is 128; the epoch is 20; and the optimization method for loss is Adam optimization.

3.2. Datasets

To test our model, we used four bridge damage datasets and compare our work with other studies that used these datasets. The experiment datasets will be described in this section.

• DIMEC-Crack Database: Lopez Droguett et al. [28] developed a new dataset for crack semantic segmentation. The dataset contains images extracted from video captured by the Unmanned Aerial Vehicle (UAV) equipped with high-resolution cameras for several concrete bridges. Each image has a resolution of 1920 x 1080; in their study, they extracted non-overlapping patches of 96 x 96 pixels from each raw image, but we will use the original raw images. The dataset contains 10,092 high-resolution images separated into two classes: 7872 crack and 2220 non-crack images.
• CODEBRIM Dataset: Mundt et al. [29] introduced a novel concrete defect bridge image (CODEBRIM) dataset. CODEBRIM includes images of 30 distinct bridges, excluding bridges without defects. The bridges were selected based on their levels of general degradation, severity, number of defects, and surface appearance, such as roughness and color. Images were captured in different weather conditions to encompass damp or soiled surfaces using numerous cameras at various magnifications. Due to the small size of the defects, high-resolution images were required. An UAV obtained a portion of the dataset due to the inaccessibility of certain bridge areas. Four distinct cameras with high resolution and huge focal lengths collected the dataset, capturing images from various distances and perspectives. Furthermore, to uniformly brighten the darkest sections of the bridge, they employed diffused flash. The additional material contains precise information about the specific camera models and their accompanying technical specifications. The collection comprises 7455 high-resolution photos of 30 distinct bridges obtained at various sizes and resolutions. We divided it into two classes: defects and background. The defects have 4388 images, while the background contains 3067 images.
• Bridge Concrete Damage (BCD): Xu et al. [19] introduced dataset, which includes 2068 images of bridge cracks and non-cracks. They captured the images using the Phantom 4 Pro's CMOS surface array camera, boasting a resolution of 1024x1024. The images underwent two reductions, first to dimensions of 512×512 and then to a size of 224×224 to produce the dataset. This dataset consists of 4057 photos with cracks and 2013 images without cracks.
• Bridge Datasets: The dataset created by Zoubir et al. [17] yielded a total of 1304 cracked and 1806 non-cracked RGB bridge images at a resolution of 200 × 200. These images depict various concrete surfaces and cracks from the actual bridge examination. In order to minimize crack-like noise, the images were pre-processed using a 3 x 3 median filter after being converted to grayscale

3.3. Evolution Metrics

Because our models are binary classification, we utilize performance metrics that experiment with the binary classification models, such as accuracy, precision, recall (also called sensitivity), f1-score, ROC-AUC score, and error rate. All these metrics are taken from the confusion matrix, which shows the classifier's performance. Four elements comprise the confusion matrix: false positive (FP), false negative (FN), true positive (TP), and true negative (TN). The formula for evaluation metrics is as follows [30,31,32]:

$$\mathrm{Accuracy}={\displaystyle \frac{\left(\mathrm{TP}+\mathrm{NT}\right)}{\left(\mathrm{TP}+\mathrm{NT}+\mathrm{FN}+\mathrm{FP}\right)}}$$

(3)

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FP}\right)}}$$

(4)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FN}\right)}}$$

(5)

$$\mathrm{F}1-\mathrm{score}=2\ast {\displaystyle \frac{\mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(6)

$$\mathrm{Error}\mathrm{Rate}={\displaystyle \frac{\left(\mathrm{FP}+\mathrm{FN}\right)}{\left(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}\right)}}$$

(7)

$$ROC-AUC={\displaystyle \frac{SP+SE}{2}}$$

(8)

$$SP={\displaystyle \frac{TN}{TN+FP}}$$

(9)

Where SP is the true negative rate (TNR), also called Specificity, and SE is the true positive rate (TPR) also called sensitivity or recall [33]. The (TP) is a true positive, (TN) is a true negative, (FP) is a false negative, and (FN) is a false negative.

3.4. Cross-Validation

For the experiment, we use 10-fold cross-validation, with 9-folds accounted for the training dataset, 1-fold for testing, and ten process repetitions. Cross-validation is essential for addressing the issue of overfitting. The selection of cross-validation folds is essential. We aggregate the outcomes of all folds by means of averaging.

4. Results and Discussion

In this section, we display a detailed analysis of the results of the proposed models on the utilized datasets. Many factors affect model performance, such as preprocessing operations and their vital role in classification performance, feature selection, dimensionality reduction, and selecting an appropriate classifier. Also, the number of features, image quality, and lighting conditions all impact classification performance. We evaluate the performance of proposed algorithm using MobileNet model. Table 2 explain the performance of the proposed model using difference evaluation metrics using four sizes of features for each dataset.

As shown in Table 2, the BCD dataset with feature sizes 100 performed the best, with an accuracy of 99.99% and a high recall of 99.99% with precision and F1 score of 100%, the ROC-AUC is 99.99% and a 0.01 error rate. The best performance was attained by the Bridge dataset with a feature size of 300, with 99.97% for all performance metrics with an error rate of 0.03. With an accuracy of 100% and a high precision of 100% in the CODEBRIM dataset, the feature size of 200 outperformed the other feature sizes. Its recall of 99.99% demonstrated its ability to identify actual damaged images with a 0.01 false negative. This resulted in an F1 score of 100%, which balances precision and recall. We also achieved a ROC-AUC of 100% and a 0-error rate. The DIMEC-Crack dataset with feature sizes 100 performed the best, with a high precision reach to 100% which means there is no false positive and 99.99% for all performance metrics with error rate reach to 0.01%. Figure 3 shows the plot of average of the best accuracy and loss. Figure 4 shows the confusion matrix of the best results for each dataset.

Table 3 presents the accuracy scores for each dataset and compares them with related studies that classify the same datasets. The difference in methodologies prevents a direct comparison with these studies, so we will only compare accuracy.

Bhalaji Kharthik et al. [15] employed 12 transfer learning (TL) methods: VGG16 achieved an accuracy of 99.83% with more than 99% for each precision, recall, and F1-score; VGG19 achieved an accuracy of 99.67% with a precision, recall, and F1-score of > 99%; Xception achieved 99.67% and more than 99% for each precision, recall, and F1-score; ResNet 50 achieved an accuracy of 99.67% with precision, recall, and F1-score reaching > 99%; the ResNet 101 achieved an accuracy of 99.5%, a precision of 100%, a recall of 99%, and an F1-score of 99.5%; the ResNet 152 gained an accuracy of 99.83% and > 99% for each precision, recall, and F1-score. InceptionV3 achieved an accuracy of 99.83%, a precision of 99%, a recall of 99%, and an f1-score of 99%. InceptionResNet V2 achieved an accuracy of 99.5% with a precision of 100%, a recall of 99%, and an F1 score reaching 99.5%. MobileNet achieved an accuracy of 99.83% with more than 99% for each precision, recall, and F1-score. MobileNetV2 achieved an accuracy of 99.83% with a precision, recall, and F1 score of > 99%. DenseNet 121 achieved an accuracy of 99.67% with a precision, recall, and F1-score of 100%, while the EfficientNetB0 model yielded an accuracy of 99.6% and a precision of 99.83%, with a precision, recall, and F1-score of > 99%.

Yang et al. [16] also used transfer learning, utilizing 13 layers of VGG16 and 2 FC layers. They trained their proposal with BCD and CCIC datasets. For the BCD dataset, they achieved an accuracy of 99.72%, a precision of 96.46%, and an AUC of 99.99%. In their study, Zoubir et al. [17] used ULBP, HOG, KPCA, and SVM to classify concrete cracks in the self-made bridge dataset. They achieved an accuracy of 99.26%, a precision of 95%, a recall of 99.23%, and an F1 score of 99.12%. Zoubir with another team [18] utilized the Bridge dataset, supplemented it with culvert images, and applied transfer learning to the CNN models. They achieved an accuracy of 94.89% in VGG16, 95.39% in VGG19, and 95.89% in InceptionV3.The InceptionV3 model produced the most favourable outcomes. The model proposed by Xu et al. [19] based on the Atrous Spatial Pyramid Pooling (ASPP) module and depthwise separable convolution. They trained the BCD dataset on 300 epochs, achieving an accuracy of 96.37% with a precision of 78.11%, 100% recall, and an F1-score of 87.71%. When we compared our proposal to previous studies and with the MobileNetV3_Large model, we found that it outperformed in all feature sizes.

5. Conclusions

This paper presents a method for enhancing bridge damage prediction based on LBP-BoVW for feature extraction, an Apriori algorithm for selecting the strongest features, and MobileNetV3_Large as a classifier. The proposed method achieved promising results for four bridge damage and crack image datasets, with an accuracy rate between 98.27% and 100% and an error rate between 1.73% and 0%. Although we trained the data using 10-fold cross-validation, which tested all dataset samples with 20 epochs, our proposed system achieved promising results compared to previous studies that used the same datasets with more training epochs and did not use cross-validation or with lower k-fold.