1. Introduction

2. Materials and Methods

2.1. Patient Cohort

Retrospectively, a series of CT images and clinical follow-up information were collected for 156 patients (comprising a total of 684 CT images) at Jiangsu Province Hospital (the first affiliated hospital with Nanjing Medical University) during the period from December 2018 to July 2022. All eligible patients underwent initial chest CT scans up to 8 weeks before commencing ICIs, with subsequent follow-up occurring at intervals of 6 to 12 weeks in accordance with National Comprehensive Cancer Network guidelines. These images were then randomly assigned to three groups: a training cohort (n=496), a validation cohort (n=64), and a test cohort (n=64). Due to practical limitations, not all patients could adhere to the standard review schedule, necessitating an extended follow-up period by 2 weeks before and after the standard review time. Additionally, external validation data were retrospectively collected from 37 patients (comprising 37 CT images and corresponding clinical follow-up data) at Nanjing Pukou People's Hospital between July 2019 and July 2023.

Clinical data, encompassing age, gender, stage, histology type, smoking status, and progression-free survival (PFS), were extracted from medical records. All patients underwent restaging in accordance with the eighth edition criteria of the American Joint Committee on Cancer [20]. Post-treatment tumour response to ICIs was evaluated following with RECIST version 1.1 [21]. Disease progression was defined as either a 20% increase in the volume of targeted lesions or the appearance of new lesions. The primary endpoint of the study is PFS, measuring the duration from the start of ICIs to either disease progression or death. Instances involving death, loss to follow-up, or absence of outcome occurrence were appropriately censored.

In the present study, patients with PFS greater than 7 months were categorized as the ICIs responded group, while those with PFS less than or equal to 7 months were classified as the ICIs non-responded group. Progression was determined through imaging reports indicating tumour growth or the appearance of new disease sites, coupled with assessments conducted by the treating physician. The compilation of clinical data was finalized for outcome analysis on September 10, 2020.

The flow chart in Figure 1 illustrates the patient selection process:

The admission criteria for patients included:

(1)

Histologically confirmed primary lung cancer.

(2)

Patients diagnosed with stage III or IV lung cancer.

(3)

Patients received first-line treatment with either ICIs monotherapy (200 mg every 3 weeks at the approved dose) or a combination of ICIs and chemotherapy.

(4)

Endpoint events and status were clearly recorded.

(5)

A full set of pre-treatment and three consecutive follow-up CT scans.

(6)

Comprehensive clinical data, encompassing gender, age, smoking status, TNM stage, pathological type, and tumour treatment regimen.

Exclusion criteria encompassed patients who fulfilled any of the subsequent criteria:

(1)

History of surgical excision before ICIs and during follow-up.

(2)

No measurable lesions in the pulmonary window according to RECIST 1.1 criteria.

Figure 1. Flow chart of patient recruitment.

Figure 1. Flow chart of patient recruitment.

2.4. Model Construction

Leveraging the concept of a global self-attention mechanism, the ViViT model was adeptly adapted to classify the treatment response of ICIs through analysis of an individual's 3D CT volume. The proposed deep learning approach is depicted in Figure 3. The methodology involves utilizing a series of 3D CT scans as input, subsequently generating binary prognosis outcomes: favourable and unfavourable. For the present analysis, 3D CT scans acquired via a CT scanner were employed.

After random data augmentation, as described in Section 4(c), the 3D CT scan series was input into the ViViT model. Initially, features of the CT scans are embedded as depicted in Figure 4. The scan series from $S\in R^{T\times H\times W\times C}$ need to be mapped to the token series $z\in R^{n_t\times n_h\times n_w\times d}$. The input series $S$ is sampled uniformly for the $n_t$ scan image, where the same tokenizing method with Vision Transformer is applied to each scan image, that is, convolution is used to downsample grids of each scan image. These tokens are concatenated to form the token series $\tilde{z}$. Additionally, due to the loss of spatial information after feeding into ViViT, learnable position embeddings are added to the token series $\tilde{z}$, and a classification token is appended at the end to capture global token features. Then, the token series $\tilde{z}$ is reshaped with position information from $R^{n_t\times n_h\times n_w\times d}$ to $R^{N\times d}$, where the first three dimensions are flattened. This completes the procedure of tokenizing the scan series.

After obtaining the token sequence of CT scan series with position embeddings, the factorized self-attention encoder is adopted to model the spatial and temporal features. As shown in Figure 5, the factorized self-attention encoder decouples the spatial and temporal self-attention heads. The self-attention weights of spatial and temporal features are calculated separately by multi-head operations. Each head module contains multiple matrices of queries $Q$, keys $K$ and values $V$, as denoted in Equations (2)–(4), which are the linear projections of $\tilde{z}$.

$$Q=W_q\tilde{z}$$

(2)

$$K=W_k\tilde{z}$$

(3)

$$V=W_v\tilde{z}$$

(4)

where the shape of $Q$, $K$ and $V$ is $R^{N\times d}$.and $N=n_t\cdot n_w\cdot n_h$.

As presented in Figure 4, in addition to the learnable query matrix $Q$, for spatial heads, the key and value of the spatial dimension are constructed, where $K_s$, $V_s\in R^{n_h\cdot n_w\times d}$. For the temporal head, the key and value of the temporal dimension are $K_t$, $V_t\in R^{n_t\times d}$. Therefore, the spatial and temporal features with self-attention mechanisms can be obtained as denoted in Equations (5) and (6):

$$Y_s=Attention\left(Q,K_s,V_s\right)$$

(5)

$$Y_t=Attention\left(Q,K_t,V_t\right)$$

(6)

where spatial and temporal features occupy half of the self-attention heads, respectively.

Finally, the features of spatial self-attention and temporal self-attention can be concatenated as denoted in Equation (7):

$$Y=Concat\left(Y_s,Y_t\right)W$$

(7)

After the operations of multi-head self-attention, layer normalization and fully connected layers are adopted as the feedforward module. Through the stack of L ViViT modules, a classification head is connected to output the result of prognosis. During the training procedure, label smoothing is adopted to alleviate the over-confidence of the model.

Figure 5. Factorized dot-product self-attention encoder.

Figure 5. Factorized dot-product self-attention encoder.

3. Results

3.4. Performance of the ViViT Model

3.4.1. ROC curve

A deep learning model was formulated to anticipate patients' likelihood of progression after immune checkpoint inhibition, employing their successive chest CT images. Following image pre-processing, the CT image data were fed into the ViViT model, thereby forecasting the efficacy of ICIs. For evaluating the predictive capabilities of the models, diverse descriptive indices such as the AUC, accuracy, precision, recall, F1 score, specificity, and confusion matrix were employed. Typically, features or models exhibiting an AUC exceeding 0.60 are widely regarded as predictive in analogous studies [23]. The ViViT model demonstrated good performance, with the AUC values for the training set, validation set, test set and external validation set being 0.74 (95% CI: 0.69 to 0.78), 0.74 (95% CI: 0.61 to 0.86), 0.76 (95% CI: 0.62 to 0.88), and 0.69 (95% CI: 0.50 to 0.87), respectively (Figure 7a–d).

3.4.2. Confusion Matrix

Further, for a more comprehensive assessment of the utility of deep learning in predicting outcomes based on patient images before and after ICIs therapy, the confusion matrix was also obtained. Figure 8a–d depict the confusion matrix of the ViViT model, portraying the outcomes of the classification between the ICIs-responded and non-responded groups. For ViViT model, the overall accuracy levels in the training set, validation set, test set, and external validation set were 0.74, 0.76, 0.75, and 0.68, precision levels were 0.80, 0.82, 0.81, and 0.63, recall levels were 0.69, 0.75, 0.72, and 0.71, F1 levels were 0.74, 0.78, 0.77, and 0.67, and specificity levels were 0.80, 0.78, 0.78, and 0.65, respectively (Table 1).

On the external validation set, the ViViT model attained an accuracy of 69%, with the AUC serving as the evaluation metric. Leveraging the evaluation metrics including AUC, accuracy, precision, recall, F1 score, and specificity, the proposed ViViT model demonstrated strong performance in the external validation set.

Table 1. Prediction performance of ViViT model.

Table 1. Prediction performance of ViViT model.

	Training set N=496	Validation set N=64	Test set N=64	External validation set N =37
AUC	0.74	0.74	0.76	0.69
Accuracy	0.74	0.76	0.75	0.68
Precision	0.80	0.82	0.81	0.63
Recall	0.69	0.75	0.72	0.71
F1	0.74	0.78	0.77	0.67
Specificity	0.80	0.78	0.78	0.65

4. Discussion

5. Conclusions

References

1. Herbst, R.S.; Morgensztern, D.; Boshoff, C. The biology and management of non-small cell lung cancer. Nature 2018, 553, 446-454. [CrossRef]
2. Shankar, B.; Zhang, J.; Naqash, A.R.; Forde, P.M.; Feliciano, J.L.; Marrone, K.A.; Ettinger, D.S.; Hann, C.L.; Brahmer, J.R.; Ricciuti, B.; et al. Multisystem Immune-Related Adverse Events Associated With Immune Checkpoint Inhibitors for Treatment of Non-Small Cell Lung Cancer. JAMA Oncol 2020, 6, 1952-1956. [CrossRef]
3. Osmani, L.; Askin, F.; Gabrielson, E.; Li, Q.K. Current WHO guidelines and the critical role of immunohistochemical markers in the subclassification of non-small cell lung carcinoma (NSCLC): Moving from targeted therapy to immunotherapy. Semin Cancer Biol 2018, 52, 103-109. [CrossRef]
4. Borghaei, H.; Paz-Ares, L.; Horn, L.; Spigel, D.R.; Steins, M.; Ready, N.E.; Chow, L.Q.; Vokes, E.E.; Felip, E.; Holgado, E.; et al. Nivolumab versus Docetaxel in Advanced Nonsquamous Non-Small-Cell Lung Cancer. N Engl J Med 2015, 373, 1627-1639. [CrossRef]
5. Shitara, K.; Özgüroğlu, M.; Bang, Y.-J.; Di Bartolomeo, M.; Mandalà, M.; Ryu, M.-H.; Fornaro, L.; Olesiński, T.; Caglevic, C.; Chung, H.C.; et al. Pembrolizumab versus paclitaxel for previously treated, advanced gastric or gastro-oesophageal junction cancer (KEYNOTE-061): a randomised, open-label, controlled, phase 3 trial. Lancet 2018, 392, 123-133. [CrossRef]
6. Yu, H.; Boyle, T.A.; Zhou, C.; Rimm, D.L.; Hirsch, F.R. PD-L1 Expression in Lung Cancer. J Thorac Oncol 2016, 11, 964-975. [CrossRef]
7. Ganesh, K.; Stadler, Z.K.; Cercek, A.; Mendelsohn, R.B.; Shia, J.; Segal, N.H.; Diaz, L.A. Immunotherapy in colorectal cancer: rationale, challenges and potential. Nat Rev Gastroenterol Hepatol 2019, 16, 361-375. [CrossRef]
8. Jia, Q.; Wu, W.; Wang, Y.; Alexander, P.B.; Sun, C.; Gong, Z.; Cheng, J.-N.; Sun, H.; Guan, Y.; Xia, X.; et al. Local mutational diversity drives intratumoral immune heterogeneity in non-small cell lung cancer. Nat Commun 2018, 9, 5361. [CrossRef]
9. van der Velden, B.H.M.; Kuijf, H.J.; Gilhuijs, K.G.A.; Viergever, M.A. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Med Image Anal 2022, 79, 102470. [CrossRef]
10. Chen, X.; Wang, X.; Zhang, K.; Fung, K.-M.; Thai, T.C.; Moore, K.; Mannel, R.S.; Liu, H.; Zheng, B.; Qiu, Y. Recent advances and clinical applications of deep learning in medical image analysis. Med Image Anal 2022, 79, 102444. [CrossRef]
11. Tian, P.; He, B.; Mu, W.; Liu, K.; Liu, L.; Zeng, H.; Liu, Y.; Jiang, L.; Zhou, P.; Huang, Z.; et al. Assessing PD-L1 expression in non-small cell lung cancer and predicting responses to immune checkpoint inhibitors using deep learning on computed tomography images. Theranostics 2021, 11, 2098-2107. [CrossRef]
12. Mu, W.; Jiang, L.; Shi, Y.; Tunali, I.; Gray, J.E.; Katsoulakis, E.; Tian, J.; Gillies, R.J.; Schabath, M.B. Non-invasive measurement of PD-L1 status and prediction of immunotherapy response using deep learning of PET/CT images. J Immunother Cancer 2021, 9. [CrossRef]
13. Yan, R.; Liao, J.; Yang, J.; Sun, W.; Nong, M.; Li, F. Multi-hour and multi-site air quality index forecasting in Beijing using CNN, LSTM, CNN-LSTM, and spatiotemporal clustering. Expert Systems with Applications 2021, 169, 114513. [CrossRef]
14. Nguyen, H.-T.; Nguyen, T.-O. Attention-based network for effective action recognition from multi-view video. Procedia Computer Science 2021, 192, 971-980. [CrossRef]
15. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30.
16. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
17. Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2021; pp. 10012-10022.
18. Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; Shao, L. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media 2022, 8, 415-424.
19. Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lučić, M.; Schmid, C. Vivit: A video vision transformer. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2021; pp. 6836-6846.
20. Amin, M.B.; Greene, F.L.; Edge, S.B.; Compton, C.C.; Gershenwald, J.E.; Brookland, R.K.; Meyer, L.; Gress, D.M.; Byrd, D.R.; Winchester, D.P. The Eighth Edition AJCC Cancer Staging Manual: Continuing to build a bridge from a population-based to a more "personalized" approach to cancer staging. CA Cancer J Clin 2017, 67, 93-99. [CrossRef]
21. Eisenhauer, E.A.; Therasse, P.; Bogaerts, J.; Schwartz, L.H.; Sargent, D.; Ford, R.; Dancey, J.; Arbuck, S.; Gwyther, S.; Mooney, M.; et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur J Cancer 2009, 45, 228-247. [CrossRef]
22. Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
23. Coroller, T.P.; Agrawal, V.; Narayan, V.; Hou, Y.; Grossmann, P.; Lee, S.W.; Mak, R.H.; Aerts, H.J.W.L. Radiomic phenotype features predict pathological response in non-small cell lung cancer. Radiother Oncol 2016, 119, 480-486. [CrossRef]
24. Kleppe, A.; Skrede, O.-J.; De Raedt, S.; Liestøl, K.; Kerr, D.J.; Danielsen, H.E. Designing deep learning studies in cancer diagnostics. Nat Rev Cancer 2021, 21, 199-211. [CrossRef]
25. Ito, K.; Schöder, H.; Teng, R.; Humm, J.L.; Ni, A.; Wolchok, J.D.; Weber, W.A. Prognostic value of baseline metabolic tumor volume measured on 18 F-fluorodeoxyglucose positron emission tomography/computed tomography in melanoma patients treated with ipilimumab therapy. European journal of nuclear medicine and molecular imaging 2019, 46, 930-939.
26. Peeken, J.C.; Bernhofer, M.; Spraker, M.B.; Pfeiffer, D.; Devecka, M.; Thamer, A.; Shouman, M.A.; Ott, A.; Nüsslin, F.; Mayr, N.A. CT-based radiomic features predict tumor grading and have prognostic value in patients with soft tissue sarcomas treated with neoadjuvant radiation therapy. Radiotherapy and Oncology 2019, 135, 187-196.
27. Tanaka, M.; Yojiro Hashiguchi, M., Hideki Ueno, MD, Kazuo Hase, MD, Hidetaka Mochizuki, MD. Tumor budding at the invasive margin can predict patients at high risk of recurrence after curative surgery for stage II, T3 colon cancer. Diseases of the colon & rectum 2003, 46, 1054-1059.
28. Mehta, T.S.; Raza, S. Power Doppler sonography of breast cancer: does vascularity correlate with node status or lymphatic vascular invasion? AJR. American journal of roentgenology 1999, 173, 303-307.
29. Trebeschi, S.; Bodalal, Z.; Boellaard, T.N.; Tareco Bucho, T.M.; Drago, S.G.; Kurilova, I.; Calin-Vainak, A.M.; Delli Pizzi, A.; Muller, M.; Hummelink, K.; et al. Prognostic Value of Deep Learning-Mediated Treatment Monitoring in Lung Cancer Patients Receiving Immunotherapy. Front Oncol 2021, 11, 609054. [CrossRef]
30. Bustos, A.; Payá, A.; Torrubia, A.; Jover, R.; Llor, X.; Bessa, X.; Castells, A.; Carracedo, Á.; Alenda, C. xDEEP-MSI: Explainable Bias-Rejecting Microsatellite Instability Deep Learning System in Colorectal Cancer. Biomolecules 2021, 11. [CrossRef]
31. Kather, J.N.; Pearson, A.T.; Halama, N.; Jäger, D.; Krause, J.; Loosen, S.H.; Marx, A.; Boor, P.; Tacke, F.; Neumann, U.P.; et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 2019, 25, 1054-1056. [CrossRef]
32. Wang, S.; Shi, J.; Ye, Z.; Dong, D.; Yu, D.; Zhou, M.; Liu, Y.; Gevaert, O.; Wang, K.; Zhu, Y.; et al. Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning. Eur Respir J 2019, 53. [CrossRef]
33. Coudray, N.; Ocampo, P.S.; Sakellaropoulos, T.; Narula, N.; Snuderl, M.; Fenyö, D.; Moreira, A.L.; Razavian, N.; Tsirigos, A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 2018, 24, 1559-1567. [CrossRef]
34. He, B.; Dong, D.; She, Y.; Zhou, C.; Fang, M.; Zhu, Y.; Zhang, H.; Huang, Z.; Jiang, T.; Tian, J.; et al. Predicting response to immunotherapy in advanced non-small-cell lung cancer using tumor mutational burden radiomic biomarker. J Immunother Cancer 2020, 8. [CrossRef]
35. Chen, Y.; Sun, Z.; Chen, W.; Liu, C.; Chai, R.; Ding, J.; Liu, W.; Feng, X.; Zhou, J.; Shen, X.; et al. The Immune Subtypes and Landscape of Gastric Cancer and to Predict Based on the Whole-Slide Images Using Deep Learning. Front Immunol 2021, 12, 685992. [CrossRef]
36. Palmeri, M.; Mehnert, J.; Silk, A.W.; Jabbour, S.K.; Ganesan, S.; Popli, P.; Riedlinger, G.; Stephenson, R.; de Meritens, A.B.; Leiser, A.; et al. Real-world application of tumor mutational burden-high (TMB-high) and microsatellite instability (MSI) confirms their utility as immunotherapy biomarkers. ESMO Open 2022, 7, 100336. [CrossRef]
37. Wang, T.; She, Y.; Yang, Y.; Liu, X.; Chen, S.; Zhong, Y.; Deng, J.; Zhao, M.; Sun, X.; Xie, D.; et al. Radiomics for Survival Risk Stratification of Clinical and Pathologic Stage IA Pure-Solid Non-Small Cell Lung Cancer. Radiology 2022, 302, 425-434. [CrossRef]
38. Li, J.; Qiu, Z.; Zhang, C.; Chen, S.; Wang, M.; Meng, Q.; Lu, H.; Wei, L.; Lv, H.; Zhong, W.; et al. ITHscore: comprehensive quantification of intra-tumor heterogeneity in NSCLC by multi-scale radiomic features. Eur Radiol 2023, 33, 893-903. [CrossRef]
39. Hochmair, M.J.; Schwab, S.; Burghuber, O.C.; Krenbek, D.; Prosch, H. Symptomatic pseudo-progression followed by significant treatment response in two lung cancer patients treated with immunotherapy. Lung Cancer 2017, 113, 4-6. [CrossRef]