1. Introduction

2. Database and Methods

2.2. Methods

The main objective of this article is to locate the scaphoid bone in X-ray images and assist doctors in determining whether it is fractured, as well as in identifying the fracture area. Displaced fractures are typically easy to detect in X-ray images, allowing doctors to quickly diagnose them. In contrast, non-displaced and occult fractures are more challenging to identify due to X-ray projection shadowing and variations in bone density among patients. Additionally, discrepancies in interpretation between radiologists and orthopedic surgeons are common. For fractures not visible in the anterior-posterior view, doctors must examine the scaphoid bone for discontinuities, making the diagnosis time-consuming and demanding a thorough understanding of the scaphoid.

Our model is designed to improve upon traditional diagnostic methods by integrating both anterior-posterior and lateral views of X-ray images. This approach provides additional side information often overlooked in standard assessments. The first stage of the model uses Faster R-CNN to extract the scaphoid bone from both views of the arm X-ray image. These extracted regions are then processed in the second stage using a multi-view fusion model developed in this study. This model separately analyzes fracture areas from both views and fuses the features to enhance discriminative capability. The fused features are used to assess whether the scaphoid bone is fractured, and the classification scores help refine the detection of fracture areas for greater accuracy. An overview of the system is illustrated in Figure 3.

Figure 3 provides a schematic of the system’s workflow. Initially, wrist X-ray images from two perspectives are collected. These images are processed to detect the scaphoid bone region, a critical component of the wrist. Once the scaphoid region is identified, the multi-view fusion model evaluates images from both perspectives to classify and locate fractures. This fusion approach aims to enhance the accuracy and reliability of fracture detection.

2.2.1. Scaphoid Bone Area Detection

As mentioned earlier, the first stage aims to extract the scaphoid bone from the entire wrist X-ray image. For this purpose, we use the Faster R-CNN model combined with a Feature Pyramid Network (FPN). Faster R-CNN is a renowned two-stage detector that incorporates a Region Proposal Network (RPN), which enhances the speed of the detector by replacing the traditional algorithmic approach used in the initial stage. The Feature Pyramid Network improves the utilization of feature maps at various scales, addressing the issue that deeper networks might lose small objects.

In our research, if the first stage fails to accurately locate the scaphoid bone, subsequent fracture detection cannot proceed effectively. Therefore, we employ the two-stage Faster R-CNN model with FPN at this stage, despite its slightly slower processing time, due to its improved accuracy. Unlike the original Faster R-CNN model, which uses a single-level feature map to generate candidate boxes, the FPN integrates feature maps from multiple levels. This approach helps retain small objects that might otherwise be lost during pooling, thereby enhancing detection performance for small and medium-sized objects.

Considering that a typical wrist X-ray image measures approximately 2000 × 1600 pixels and the scaphoid bone region is around 150 × 150 pixels, the scaphoid falls into the category of small and medium-sized objects. To address this, we use the Feature Pyramid Network (FPN) which upsamples and enlarges deep features to match the size of previous layers, then integrates them using a convolutional layer. Specifically, after incorporating the FPN, a 1×1 convolutional layer adjusts the channel number to 256, and deep features are upsampled and combined with the previous layer’s features. A final 3×3 convolutional layer refines the feature map. This multi-layer approach, with varied anchor box sizes and aspect ratios, enhances overall detection performance.

Additionally, the Faster R-CNN model in this study uses RoI Align to generate feature maps of consistent dimensions for different proposals. RoI Align addresses the limitations of RoI Pooling by avoiding rounding issues associated with dividing regions into smaller blocks. Instead, RoI Align performs bilinear interpolation on these blocks, yielding more accurate feature maps and improving detection results.

To handle multiple bounding boxes that may overlap, Non-Maximum Suppression (NMS) is applied to retain the most accurate boxes. NMS starts by selecting the bounding box with the highest score and storing the remaining boxes in an output list. It then calculates the Intersection over Union (IoU) between the selected box and the remaining boxes. If the IoU exceeds a predefined threshold, the overlapping box is removed from the list. This process repeats until only the boxes with the highest scores remain, effectively representing the detected objects.

2.2.2. Fracture Classification and Detection

The following sections will detail the multi-view fusion model proposed in this article. Multi-view image techniques have shown promise in enhancing classification capabilities, particularly in clinical settings where accurately locating fractures is crucial. Effective fusion of multi-view data is essential to improving fracture detection beyond what can be achieved with 2D projection images alone. The model presented consists of four key components: the backbone network, the context extractor, the Feature Pyramid Network (FPN), and the multi-view fusion module, as illustrated in Figure 4.

Backbone Network: The anterior-posterior (AP) and lateral (LA) views are first processed through the backbone network to extract features. The backbone network must have a sufficiently large receptive field to capture meaningful features. Deeper networks like ResNet, which use residual connections to mitigate issues like gradient vanishing, are ideal. ResNet-152 was found to be the most effective in our experiments among ResNet-50, ResNet-101, and ResNet-152. It provides deep, high-quality features, with the last three blocks of ResNet-152 feeding into the context extractors.

Context Extractor: To address the challenge of detecting occult or non-displaced fractures, which can be camouflaged due to lighting or bone density variations, the context extractor expands the receptive field without pooling, using dilated convolutions. This approach, as demonstrated by Mei et al. [34], helps in extracting rich semantic features across a broader context. The context extractor is designed with four branches (as shown in Figure 5):

Each context extractor contains four context extraction branches, as shown in Figure 5. First, the $1\times 1$ convolution compresses the number of channels to $1/4$ of the original and then uses the $k_i\times k_i$ convolution and the$3\times 3$ dilated convolution with a dilation rate of $r_i$ to extract context-aware features; where $i$ represents the context extraction branch of $i^{th};$ $k_i\in \left[\mathrm{1,3},\mathrm{5,7}\right],r_i\in \left[\mathrm{1,2},\mathrm{4,8}\right]$. Each convolutional layer is followed by a set of batch normalization layers and a ReLU. The output of $i^{th}$ is added with the output of ${\left(i+1\right)}^{th}$ to stack the different ranges of receptive fields together. Finally, the outputs of the four context extraction branches are concatenated together and use $1\times 1$ convolution to integrate all features. Such a design enables the context extractor to obtain a wide range of rich perceptual features, enabling the model to have contextual reasoning capabilities and to see finer differences in feature maps.

Feature Pyramid Network (FPN): The FPN enhances the feature maps extracted from different scales by combining deep and shallow features, improving detection of small and medium-sized objects like the scaphoid bone.

Multi-View Fusion Module: After feature extraction and context enhancement, the FPN outputs from the AP and LA views are fed into the multi-view fusion module. This module integrates features from both views to enhance fracture classification and detection accuracy. The multi-view fusion module uses the FPN features from the AP views and LA views for fusion, and the feature maps of the corresponding layers are input into the fusion block, as shown in Figure 6. The fusion block first performs global average pooling on the feature maps of the two views and outputs [B, C, 1, 1] to flatten the feature maps and concatenate them. Then the weight transform designed calculates the common important information of the two views, and then multiply this weight back to the respective views. Finally, the features of the two views are concatenated together for the final classification. The goal is to find as much of the fractured scaphoid region as possible, we add the output of $i^{th}$ to the output of ${\left(i+1\right)}^{th}$, where $i$ represents the ith fusion block. First use the global average pooling layer to flatten and concatenate the corresponding feature pyramid layers of two different views, then use the weight transform layer to calculate the common information of different views, and finally multiply the weights back to the features of different views and concatenate them.

Figure 6. Multi-view fusion module architecture.

Figure 6. Multi-view fusion module architecture.

Algorithm:

1) Feature Extraction: Both AP and LA views are processed through the backbone network to extract initial features.

2) Contextual Information: The context extractor refines these features by expanding the receptive field and integrating contextual information.

3) Feature Fusion: The FPN combines features from multiple scales to improve detection performance.

4) Final Classification and Detection: The multi-view fusion module combines features from both views to classify fractures and re-evaluate detection boxes for precise localization.

The model aims to improve the accuracy of fracture detection, particularly for occult and non-displaced fractures, by leveraging comprehensive multi-view information and advanced feature extraction techniques.

Most of the detection methods, regardless of one-stage or two-stage methods, only use a single scale for prediction. However. in order to extract better features, the depth of the model is often very deep that the loss of information on small objects. FPN can well combine the deep and shallow information, so that the feature map of each scale can be better used, and greatly improve the detection results of small objects. The ground-truth of the fractures in this article also belong to small objects, so a FPN is added after the context extractor. The structure here is the same as Faster R-CNN. First, the $1\times 1$ convolution layer is used to adjust the number of channels of the input feature map to $256$, and then the $i^{th}$ feature map is upsampled and added with ${\left(i-1\right)}^{th}$ feature maps, and finally the information is integrated through a $3\times 3$ convolution.

Figure 7. Fusion block architecture in the multi-view fusion module.

Figure 7. Fusion block architecture in the multi-view fusion module.

Weight transform is mainly used to combine information from AP views and LA views. It is too rough to classify the features directly after merging as mentioned above. In general, the importance of the AP view is higher than that of the LA view. Therefore, classification after direct concatenation may cause all the weights to be pulled away by the AP views and results in the loss of the features of the LA views. Therefore, we design a weight transform, which combines the features of the AP views and the LA views, calculates the common important information of the two views, and then multiplies them back to their respective views. In this way, the information of the LA views is preserved and avoided its weight being completely discarded. Equations (1) and (2) show the module construction by different feature map AP_i and LP_i of scale i.

$$MultiviewFusionModule=\sum _{i=0}^{l}C(A{P}_i\times W_i,L{A}_i\times W_i)$$

(1)

$$W_i=Sigmoid\left(L\right(ReLU\left(L\left(C\left(GAP\left(A{P}_i\right),GAP\left(L{A}_i\right)\right)\right)\right)\left)\right)$$

(2)

where $l$ represents the number of layers of the feature pyramid; $L$ represents the fully connection layer; $C$ represents the concatenation, and the number of channels output by the two fully connected layers is $1/2$ of the number of input channels.

The architecture of the detection branch in the second stage is a fully convolutional network. The classification of the detection branch and the prediction head of the detection is the same that consists of $3\times 3$ convolutional layers, batch normalization layer, ReLU, and $3\times 3$ convolutional layers. The number of output channels of the classification branch is $A\times C$, and the number of channels of the detection branch is$A\times 5$. $A$ and $C$ represent the number of anchor boxes and categories, respectively, and $5$ represents $(x,y,w,h,\theta )$, representing the center point coordinates, width, high, and angle of the prediction box [22]. Each layer of the FPN has its prediction head, and finally, the results of each layer are merged to unify match with ground-truth and calculate the loss. The multi-view fusion classification branch consists of a fully connected layer, ReLU and a fully connected layer. The first fully connection layer input the refined features by the multi-view fusion module, and the output channel is half of the number of inputs. The output of the second fully connection layer is 2, indicating the probability of fracture and non-fracture.

To pass information to fracture detection in two different views through this classification branch, the classification results are re-scored the prediction boxes in the final prediction stage. In the most perfect case, both the classification score and the predicted box score will be close to 1, so when both branch predictions are at their highest, the sum is 2 and the smallest is 0. We take 0.5 as the threshold of the classification branch to keep the classification score greater than or equal to the threshold and set the classification score to 0 if it is smaller than the threshold. The purpose of this is to give priority to the classification results. Because the classification branch is directly using the multi-view feature, it can better distinguish whether the scaphoid is fractured or not. The formula of rescore is shown as Equation (3).

$$Rescore=Threshold\left({Score}_{cls},0.5\right)+{Score}_{det}$$

(3)

The model in the second stage consists of three parts, namely the detector for AP views, the detector for LA views, and the classifier for multi-view fusion. The following will describe their loss functions respectively. The detector part is similar to Faster R-CNN, but regression an additional angle. First encode the ground truth box $(x,y,w,y,\theta )$and prediction box $(x^{⬚},y^{⬚},w^{⬚},h^{⬚},{\theta }^{⬚})$ as $v=(t_x,t_y,t_w,t_h,t_{\theta })$ and $v^{⬚}=(t_x^{⬚},t_t^{⬚},t_w^{⬚},t_h^{⬚},t_{\theta }^{⬚})$, anchor box is expressed as $\left(x_a,y_a,w_a,h_a,{\theta }_a\right)$. $v$ and $v^{⬚}$ are defined as Equations (4) and (5):

$$\begin{gathered} t_x={\displaystyle \frac{x-x_a}{w_a}},t_y={\displaystyle \frac{y-y_a}{h_a}} \\ {t}_w=log\left({\displaystyle \frac{w}{w_a}}\right),t_h=log\left({\displaystyle \frac{h}{h_a}}\right) \\ {t}_{\theta }={\displaystyle \frac{4\theta }{\pi }} \end{gathered}$$

(4)

$$\begin{gathered} t_x^{⬚}={\displaystyle \frac{x^{⬚}-x_a}{w_a}},t_y^{⬚}={\displaystyle \frac{y^{⬚}-y_a}{h_a}} \\ {t}_w^{⬚}=log\left({\displaystyle \frac{w^{⬚}}{w_a}}\right),t_h^{⬚}=log\left({\displaystyle \frac{h^{⬚}}{h_a}}\right) \\ {t}_{\theta }^{⬚}=tanh({\theta }^{⬚}) \end{gathered}$$

(5)

And use the $SmoothL1$ loss to regress the positive samples, the $SmoothL1$ loss is defined in Equation (6):

$$L_{reg}={\displaystyle \frac{1}{N_{pos}}}\sum _1^{N_{pos}}Smoot{h}_{L1}(v_i^{⬚}-v_i)$$

(6)

The classification prediction head of the detection branch uses the Binary Cross Entropy loss as Equations (7) and (8):

$$L_{cls}={\displaystyle \frac{1}{N_{pos}}}\sum _{i=1}^{N}BCE(y_i,p_i)$$

(7)

$$BCE\left(y,p\right)=\left\{\begin{array}{c}-log\left(p\right)ify=1\\ log\left(1-p\right)ify=0\end{array}\right.$$

(8)

Since the number of negative samples is much larger than that of positive samples, we divide the number of positive samples rather than the total number of samples to take the average, which can avoid the classifier from being affected by too many negative samples and can better solve the problem of sample imbalance. problems to improve the effectiveness of training. $WhereN$ represents the number of total samples, and $N_{pos}$ represents the number of positive samples. The loss function of multi-view fusion classification is the same as that of Faster R-CNN, and both use the Cross Entropy loss. The total loss is defined as Equation (9):

$$L=L_{A{P}_{cls}}+L_{A{P}_{reg}}+L_{L{A}_{cls}}+L_{L{A}_{reg}}+L_{multi-view\_cls}$$

(9)

3. Results

4. Discussion

5. Conclusions

References

1. Gibney, B., et al., Incorporating cone-beam CT into the diagnostic algorithm for suspected radiocarpal fractures: a new standard of care? American Journal of Roentgenology, 2019. 213(5): p. 1117-1123. [CrossRef]
2. Bochkovskiy, A., C.-Y. Wang, and H.-Y.M. Liao, Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020.
3. Lin, T.-Y., et al. Focal loss for dense object detection. in Proceedings of the IEEE international conference on computer vision. 2017.
4. Tian, Z., et al. Fcos: Fully convolutional one-stage object detection. in Proceedings of the IEEE/CVF international conference on computer vision. 2019.
5. Ren, S., et al., Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 2015. 28.
6. Cai, Z. and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
7. He, K., et al. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
8. Huang, G., et al. Densely connected convolutional networks. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
9. Tan, M. and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. in International conference on machine learning. 2019. PMLR.
10. Liu, W., et al. Ssd: Single shot multibox detector. in European conference on computer vision. 2016. Springer.
11. Kong, T., et al., Foveabox: Beyound anchor-based object detection. IEEE Transactions on Image Processing, 2020. 29: p. 7389-7398. [CrossRef]
12. Qian, Y., et al. R-CNN Object Detection Inference With Deep Learning Accelerator. in 2018 IEEE/CIC International Conference on Communications in China (ICCC Workshops). 2018. IEEE.
13. He, K., et al. Mask r-cnn. in Proceedings of the IEEE international conference on computer vision. 2017.
14. Yadav, D. and S. Rathor. Bone fracture detection and classification using deep learning approach. in 2020 International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC). 2020. IEEE.
15. Raisuddin, A.M., et al., Critical evaluation of deep neural networks for wrist fracture detection. Scientific projects, 2021. 11(1): p. 1-11.
16. Hu, J., L. Shen, and G. Sun. Squeeze-and-excitation networks. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
17. Yoon, A.P., et al., Development and validation of a deep learning model using convolutional neural networks to identify scaphoid fractures in radiographs. JAMA network open, 2021. 4(5): p. e216096-e216096. [CrossRef]
18. Selvaraju, R.R., et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. in Proceedings of the IEEE international conference on computer vision. 2017.
19. Lindsey, R., et al., Deep neural network improves fracture detection by clinicians. Proceedings of the National Academy of Sciences, 2018. 115(45): p. 11591-11596.
20. Chen, H., et al. Anatomy-aware siamese network: Exploiting semantic asymmetry for accurate pelvic fracture detection in x-ray images. in European Conference on Computer Vision. 2020. Springer.
21. Lee, C., et al., Classification of femur fracture in pelvic X-ray images using meta-learned deep neural network. Scientific projects, 2020. 10(1): p. 1-12.
22. Tai-HuaYang, Ming-Huwi Horng, Rong-Shiang Li and Yung-Nien Sun. Scaphoid Fracture Detection by Using Convolutional Neural Network, Diagnostics 2022, 12(4), 895. [CrossRef]
23. Luo, K., et al. Attention-aware multi-view stereo. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
24. Lin, T.-Y., et al. Microsoft coco: Common objects in context. in European conference on computer vision. 2014. Springer.
25. Yu, F. and V. Koltun, Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
26. Peng, S., et al. Multi-View Weighted Feature Fusion Using CNN for Pneumonia Detection on Chest X-Rays. in 2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM). 2021. IEEE.
27. Khan, H.N., et al., Multi-view feature fusion based four views model for mammogram classification using convolutional neural network. IEEE Access, 2019. 7: p. 165724-165733. [CrossRef]
28. Li, Y., et al., Automated Multi-View Multi-Modal Assessment of COVID-19 Patients Using Reciprocal Attention and Biomedical Transform. Frontiers in public health, 2022. 10.
29. Wang, Y., et al. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. in Conference on Robot Learning. 2022. PMLR.
30. Wang, G., et al., Multi-view adaptive fusion network for 3d object detection. arXiv preprint arXiv:2011.00652, 2020.