1. Introduction

Figure 1. Clinical Workflow.

Figure 1. Clinical Workflow.

2. Materials and Methods

2.2. Data

2.2.1. MSD

The MSD Task06_Lung was used for the training of the segmentation model. The dataset consisted of ninety-six thin-section (<1.5mm) positive samples from different patients with a corresponding annotated file that outlined the tumor. The patients had non-small-cell lung cancer and the mask file was marked by an expert radiologist. Out of the ninety-six files, thirty-two were marked for testing and had no corresponding mask file (dataset was originally used for a competition), leaving sixty-four to train on.

Figure 2. View of original 3D CT scan with its corresponding mask from MSD dataset.

Figure 2. View of original 3D CT scan with its corresponding mask from MSD dataset.

2.2.2. LUCAS

There were a total of 830 scans with 72 of them being positive (8.5%). The scans were taken in a clinical setting and were diagnosed by an expert physician. Many of the patients had respiratory diseases or had a high risk of developing lung cancer. Of the 830 scans, 20% (166) were set aside for testing. Due to the heavy imbalance in data, the choice was made to balance the dataset and use an equal amount of positive and negative files, leading to 104 training files. The biomarkers contained 77 different values which covered patient information, visual information, and internal markers.

Figure 3. Correlation of biomarkers with Cancer.

Figure 3. Correlation of biomarkers with Cancer.

2.3. Preprocessing

2.3.1. MSDL

Each Nifti (NII) file was split along the z-axis into 300-700 slices at 512x512 resolution using the med2image library. Conversion of all scans resulted in 17,000 slices, 90% of which did not contain a visible tumor in the corresponing mask. Data was filtered to only keep those with the tumor, resulting in 1756 files. The original dataset (In 3D) was 13 gigabytes, and the new dataset was 356 megabytes (36.5x smaller).

Figure 4. Example training image with its corresponding mask.

Figure 4. Example training image with its corresponding mask.

2.3.2. LUCAS

All images were resized to 256x256x256 (height, width, slices) to reduce compute processing needs and create consistency. A Z-score normalization was applied to all slices to ensure internal consistency of each image. Finally, if any images were below the necessary size, they were symmetrically padded.

2.4. Model Architecture

2.4.1. Segmentation Model

The segmentation model architecture used was DeepLabv3-ResNet-101. This specific architecture was used as it is one of the best performing models on the Pascal-Voc test [12]. It has also proven effective in working with lung CT scans as shown by Polat [13]. The model utilizes ResNet-101 as a basic feature extractor which then gets passed on to both the decoder and the Atrous convolution pyramid shown in Figure 5 for higher level features. The decoder concatenates the ResNet and the deeper level features while upscaling them to generate the segmentation map.

Figure 5. DeepLabv3 full architecture.

Figure 5. DeepLabv3 full architecture.

2.4.2. Biomarker Architecture

The biomarker information is passed through a multilayer perceptron (MLP) which weighs each of the 77 features and produces a singular score. This score is passed through a sigmoid to obtain a probability and ensure consistency between itself and the imaging model.

Figure 6. Biomarker MLP.

Figure 6. Biomarker MLP.

2.4.3. Multimodal Architecture

The imaging-biomarker multimodal model takes in two inputs, a probability from the imaging section and one from the biomarkers. For the imaging half, the full CT scan is used but is internally split up into slices and normalized. They are then passed to the trained version of Deeplabv3 to produce segmentation maps. The maps are then passed through a threshold system which averages the confidence of all pixels believed to be positive, which is then returned to the rest of the multimodal model. This confidence is then combined with the biomarker confidence which was simultaneously being calculated, and then passed through a fully connected layer for a single final output.

Figure 7. Multimodal model.

Figure 7. Multimodal model.

2.5. Model Training

Table 1. Hardware specifications .

Table 1. Hardware specifications .

Part	Specification
CPU	2x vCPU
GPU	Nvidia A100 40GB
RAM	32 GB
Storage	Google Drive

2.5.1. Segmentation Model Training

The model was trained for twenty-five epochs with a batch size of sixteen. It used Binary-Cross-Entropy loss and Adam optimizer. A 0.80.2 cross validation was used for evaluation (1405 and 351 files respectively).

Table 2. Segmentation model metrics .

Table 2. Segmentation model metrics .

Metric	Calculation
Specificity	${\displaystyle \frac{2XAreaofOverlap}{TotalArea}}$
Dice Coefficient	${\displaystyle \frac{TrueNegative}{TrueNegative+FalsePositive}}$

2.5.2. Multimodal Model Training

The model was trained with fifteen epochs and a batch size of four. It used Binary Cross Entropy with logits loss, Adam Optimizer with a weight decay of 1e - 5, AMSgrad active and patience of three. The model was tested on 166 set aside files from the original dataset containing 13 positive files.

Table 3. Multimodal model metrics.

Table 3. Multimodal model metrics.

Metric	Calculation
F1 score	${\displaystyle \frac{2XPrecision*Recall}{Precision+Recall}}$
Average Precision	${\displaystyle \sum _n}(R_n-R_{n-1})P_n$

3. Results

3.1. Segmentation Results

The results produced by this approach provides evidence of the accuracy of 2D segmentation compared to 3D as seen through the test DICE coefficient of 91.4%, and Specificity of 99.9%.

Figure 8. Model Loss and metrics during training.

Figure 8. Model Loss and metrics during training.

3.3. Model Comparasion

Table 4. Comparison of image models (general).

Table 4. Comparison of image models (general).

Author	Accuracy/Dice	Model Type	Data Size
Agnes et al.	0.83	3D Segmentation	300
S. Primakov et al.	0.82	3D segmentation	1328
L. Daza et al.	0.207	3D classification	830
S. Zhang et al.	0.97	2D classification	1018
M. Roa et al.	0.191	3D classification	830
Us	0.91	2D segmentation	64

Table 5. Comparison of multimodal models on LUCAS.

Table 5. Comparison of multimodal models on LUCAS.

Author	ROC	Precision	F1
J. Barrett et al.	0.847	0.53	0.508
L. Daza et al.	0.72	0.09	0.25
M. Roa et al.	-	0.251	0.341
Us	0.89	0.91	0.85

4. Discussion

Figure 9. Multimodal metrics during training.

Figure 9. Multimodal metrics during training.

Figure 10. My model’s output on new augmented data compared to the ground truth.

Figure 10. My model’s output on new augmented data compared to the ground truth.

5. Limitations

6. Conclusion

References

1. Rampinelli C, Origgi D, Bellomi M. Low-dose CT: technique, reading methods and image interpretation. Cancer Imaging. 2013 Feb 8;12(3):548-56. [CrossRef] [PubMed] [PubMed Central]
2. Parsons AM, Ennis EK, Yankaskas BC, Parker LA Jr, Hyslop WB, Detterbeck FC. Helical computed tomography inaccuracy in the detection of pulmonary metastases: can it be improved? Ann Thorac Surg. 2007 Dec;84(6):1830-6. [CrossRef] [PubMed]
3. Peuchot M, Libshitz HI. Pulmonary metastatic disease: radiologic-surgical correlation. Radiology. 1987 Sep;164(3):719-22. [CrossRef] [PubMed]
4. Abdullah Muzahid, Wan, W., & Hou, L. (2020). A New Volumetric CNN for 3D Object Classification Based on Joint Multiscale Feature and Subvolume Supervised Learning Approaches. Computational Intelligence and Neuroscience, 2020, 1–17. [CrossRef]
5. Kline, A., Wang, H., Li, Y. et al. Multimodal machine learning in precision health: A scoping review. npj Digit. Med. 5, 171 (2022). [CrossRef]
6. Primakov, S.P., Ibrahim, A., van Timmeren, J.E. et al. Automated detection and segmentation of non-small cell lung cancer computed tomography images. Nat Commun 13, 3423 (2022). [CrossRef]
7. Zhang S, Sun F, Wang N, Zhang C, Yu Q, Zhang M, Babyn P, Zhong H. Computer-Aided Diagnosis (CAD) of Pulmonary Nodule of Thoracic CT Image Using Transfer Learning. J Digit Imaging. 2019 Dec;32(6):995-1007. [CrossRef] [PubMed] [PubMed Central]
8. Daza, L., Castillo, A., Escobar, M., Valencia, S., Pinzón, B., Arbeláez, P. (2020). LUCAS: LUng CAncer Screening with Multimodal Biomarkers. In: Syeda-Mahmood, T., et al. Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures. CLIP ML-CDS 2020 2020. Lecture Notes in Computer Science(), vol 12445. Springer, Cham. [CrossRef]
9. Roa, M., Daza, L., Escobar, M., Castillo, A., Arbelaez, P. (2021). SAMA: Spatially-Aware Multimodal Network with Attention For Early Lung Cancer Diagnosis. In: Syeda-Mahmood, T., et al. Multimodal Learning for Clinical Decision Support. ML-CDS 2021. Lecture Notes in Computer Science(), vol 13050. Springer, Cham. [CrossRef]
10. Barrett, J.; Viana, T. EMM-LC Fusion: Enhanced Multimodal Fusion for Lung Cancer Classification. AI 2022, 3, 659-682. [CrossRef]
11. Antonelli, M., Reinke, A., Bakas, S. et al. The Medical Segmentation Decathlon. Nat Commun 13, 4128 (2022). [CrossRef]
12. ‌Singh, V. (2023, November 6). Deeplabv3 & Deeplabv3+ The Ultimate Pytorch Guide. LearnOpenCV. https://learnopencv.com/deeplabv3-ultimate-guide/.
13. Polat, H.: A modified DeepLabV3+ based semantic segmentation of chest computed tomography images for COVID-19 lung infections. Int. J. Imaging Syst. Technol. 32(5), 1481–1495 (2022). [CrossRef]
14. Chen, L., Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. ArXiv. /abs/1706.05587.
15. Boehm, K.M., Khosravi, P., Vanguri, R. et al. Harnessing multimodal data integration to advance precision oncology. Nat Rev Cancer 22, 114–126 (2022). [CrossRef]
16. Geert Litjens, Kooi, T., Babak Ehteshami Bejnordi, Arnaud, Ciompi, F., Mohsen Ghafoorian, van, Bram van Ginneken, & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. [CrossRef]
17. Merino, I., Azpiazu, J., Remazeilles, A., & Sierra, B. (2021). 3D Convolutional Neural Networks Initialized from Pretrained 2D Convolutional Neural Networks for Classification of Industrial Parts. Sensors, 21(4), 1078. [CrossRef]
18. Yu, J., Yang, B., Wang, J., Leader, J., Wilson, D., & Pu, J. (2020). 2D CNN versus 3D CNN for false-positive reduction in lung cancer screening. Journal of Medical Imaging, 7(05). [CrossRef]
19. Ali Ahmed, S. A., Yavuz, M. C., Şen, M. U., Gülşen, F., Tutar, O., Korkmazer, B., Samancı, C., Şirolu, S., Hamid, R., Eryürekli, A. E., Mammadov, T., & Yanikoglu, B. (2022). Comparison and ensemble of 2D and 3D approaches for COVID-19 detection in CT images. Neurocomputing, 488, 457–469. [CrossRef]
20. ‌Geert Litjens, Kooi, T., Babak Ehteshami Bejnordi, Arnaud, Ciompi, F., Mohsen Ghafoorian, van, Bram van Ginneken, & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. [CrossRef]
21. Boehm, K.M., Aherne, E.A., Ellenson, L. et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer 3, 723–733 (2022). [CrossRef]
22. Zhang, G., Yang, Z. & Jiang, S. Automatic lung tumor segmentation from CT images using improved 3D densely connected UNet. Med Biol Eng Comput 60, 3311–3323 (2022). [CrossRef]
23. Agnes SA, Anitha J. Efficient multiscale fully convolutional UNet model for segmentation of 3D lung nodule from CT image. J Med Imaging (Bellingham). 2022 Sep;9(5):052402. Epub 2022 May 11. [CrossRef] [PubMed] [PubMed Central]
24. Singh, S.P.; Wang, L.; Gupta, S.; Goli, H.; Padmanabhan, P.; Gulyás, B. 3D Deep Learning on Medical Images: A Review. Sensors 2020, 20, 5097. [CrossRef]
25. Yu, H., Li, J., Zhang, L. et al. Design of lung nodules segmentation and recognition algorithm based on deep learning. BMC Bioinformatics 22 (Suppl 5), 314 (2021). [CrossRef]
26. Voigt, W.; Prosch, H.; Silva, M. Clinical Scores, Biomarkers and IT Tools in Lung Cancer Screening—Can an Integrated Approach Overcome Current Challenges? Cancers 2023, 15, 1218. [CrossRef]