Prerpints.org logo
Preprint
Review

Pet Radiomics and Response to Immunotherapy in Lung Cancer: A Systematic Review of the Literature

Altmetrics

Downloads

106

Views

54

Comments

0

A peer-reviewed article of this preprint also exists.

Submitted:

09 May 2023

Posted:

11 May 2023

You are already at the latest version

Alerts
Abstract
Aim: to provide a comprehensive overview of the existing literature concerning the applications of positron emission tomography (PET)-radiomics in lung cancer patients candidates or undergoing immunotherapy. Materials and Methods: A systematic review was conducted on databases and web sources. English-language original articles were considered. The title and abstract were in-dependently reviewed to evaluate study inclusion. Papers duplicate, out-of-topic, review or edi-torials articles and letters to editors were excluded. For each study, the radiomics analysis was assessed based on the relies on radiomics quality score (RQS 2.0). The review was registered on the PROSPERO database with the number CRD42023402302. Results: 15 papers were included, 13 were qualified as conventional radiomics approaches, and two were as Deep Learning radiomics. The content of each study was different, indeed, 7 papers investigated the potential role of radiomics to predict PD-L1 expression and tumor microenvironment before starting immunotherapy. Moreo-ver, 2 were relative to the prediction of response and 4 investigated the utility of radiomics to predict the response to immunotherapy. Finally, 2 papers were relative to the prediction of adverse events due to the immunotherapy. Conclusions: radiomics is promising in the evaluation of TME and for the prediction of response to immunotherapy, but some limitations should be overpassed.
Keywords: 
Subject: Medicine and Pharmacology  -   Oncology and Oncogenics

1. Introduction

Lung cancer (LC) is the leading cause of cancer-related death worldwide and represents a serious threat for public health [1] despite advances in diagnosis and therapy [2,3]. Surgical resection is the standard of care for LC patients at stages I and II, while, for many years, platinum-based chemotherapy has represented a mainstay for the management of patients with an extensive disease [4]. Recently, the therapeutic landscape has been thoroughly changed by the implementation of immune-checkpoint inhibitors (ICIs).
In metastatic non-small cell lung cancer (NSCLC) harboring driver mutations (e.g., EGFR, ALK or ROS1), targeted therapies are usually preferred over other approaches, since they have been found effective with tolerable toxicity. Nevertheless, even in the absence of driver mutations, ICIs should be considered alone or in combination with chemotherapy, as a valuable option [4]. In 2015, a phase-III comparative study showed that nivolumab, a monoclonal antibody directed towards PD-1, provided a significant benefit in terms of prolonged overall survival (OS), with respect to docetaxel, in squamous and non-squamous NSCLC submitted to immunotherapy as a second-line regimen [5]. These initially encouraging results were further confirmed in subsequent clinical trials, leading to the implementation of ICIs for the management of patients with advanced NSCLC [6]. Also in SCLC, randomized phase III trials demonstrated that the combination of the ICIs, such as atezolizumab or durvalumab with platinum-etoposide chemotherapy led to an improvement of OS in patients with an extensive disease [7]. However, it has to be underlined that only 50% of LC patients will show a response to immunotherapy [8]. Furthermore, immune-related adverse events (irAEs), namely the side effects of ICIs treatment potentially occurring in any organ or system with a wide spectrum of severity, affect up to 76% of the patients under immunotherapy and can represent an important cause of treatment discontinuation [9]. From this perspective, there is an unmet need for laboratory and imaging biomarkers suitable for identifying LC patients who are more likely to benefit from ICIs.
Positron emission tomography/computed tomography (PET/CT) with fluorine-18-fluorodeoxyglucose ([18F]FDG) has a well-established role in staging and response assessment in many oncological conditions [10]. As concerns patients’ prognostic stratification before immunotherapy, [18F]FDG PET/CT has been applied with interesting preliminary results, since some PET-derived parameters, such as whole-body metabolic tumor volume (wbMTV) and total lesion glycolysis (wbTLG), were found predictors of response [11,12].
In recent years, radiomics, an emerging discipline based on the quantitative analysis of imaging data, has been gaining ever-increasing attention for its capability through machine-learning analysis to generate predictive models [13]. The main scope of radiomics is to extract, from medical images, quantitative data (i.e., features), undetectable to the human eye, that should be reproducible, interpretable and correlated with some clinical endpoints (e.g., response to therapy, treatment failure, survival, etc…). In this regard, pre-clinical and clinical studies suggest the potential of PET-radiomics for the prediction of immunotherapy response [14,15,16].
The aim of the present systematic review is to provide a comprehensive overview of the existing literature concerning the applications of PET-radiomics in LC patients candidates or undergoing ICIs, trying to outline the most relevant issues emerging from data analysis and delineate potential next steps for its widespread implementation in clinical practice.

2. Materials and Methods

2.1. Research Strategy and Study Selection

A systematic review was conducted in accordance with the preferred reporting items for systematic reviews guidelines (PRISMA) by R.L., L.U. and P.G. [17]. The authors ran queries to retrieve prospective or retrospective studies on the use of radiomics application on PET images of immunotherapy applied to lung cases on databases and web sources (i.e., PubMed, Google Scholar, and Scopus). The search was done on 12th March 2023, using multiple queries here reported: “(lung cancer OR NSCLC) AND (immune checkpoint inhibitors OR immunotherapy OR ICI) AND (radiomics OR features) AND pet NOT review”, “pet AND (radiomic OR radiomics OR texture) AND lung AND immunotherapy”, “pet AND (radiomics OR texture) AND lung AND ICI”,” pet AND (artificial intelligence OR deep learning) AND lung AND immunotherapy”,” pet AND (radiomic OR deep learning) AND lung”. English-language original articles were considered. The review was registered on the PROSPERO database with the number CRD42023402302.
The title and abstract were independently reviewed by three authors (P.G., L.E., L.F.) to evaluate study inclusion. Full articles were retrieved when the abstract was considered relevant. Papers duplicate, out-of-topic, review or editorials articles and letters to editors were excluded.

2.2. Radiomics Methodology and Study Quality

Radiomics approaches were divided into two groups: the conventional ones (also referred to as ‘hand-crafted’) and those based on Deep Learning [18]. Conventional radiomics involved the delineation of the region of interest (ROI) (which can be manual, semi-automated or fully automated) and the subsequent extraction of a set of pre-defined parameters such as first-order statistics, shape and texture features [19]. The features are eventually fed to some classification and/or regression model to produce predictions about the clinical endpoint investigated. By contrast, Deep Learning radiomics made use of computational architectures (Convolutional Neural Networks – CNNs) in which the features are no longer defined a priori but learned from the data. In this scheme, the detailed lesion delineation is not strictly necessary, as this step is replaced by approximate localization, which is generally achieved by defining a fixed-shape (typically square) bounding box around the suspicious area. Furthermore, CNNs have internal classification blocks (fully connected layers) which makes external classifiers unnecessary.
For each study, the radiomics analysis was assessed based on the relies on radiomics quality score (RQS 2.0 https://www.radiomics.world/rqs2)) introduced by Lambin and colleagues in 2017 [20] to specifically evaluate the quality of reporting in the radiomics context. RQS 2.0 consists of 36 checkpoints that reward or penalize radiomics studies to encourage best scientific practices. For a robust calculation, RQS 2.0 was assessed by a multidisciplinary panel of three raters– respectively, one nuclear medicine physician (F.F.), one medical physicist (L.M.) and one engineer (F.B.) – all with at least 5 years of experience in radiomics. After a preliminary training session to calibrate the methodology and the scoring system, each rater read, assessed and scored the papers independently. Once this step was completed, the evaluation panel reconvened for a final joint session in which a consensus score was assigned to each paper.

3. Results

3.1. Radiomics Assessment

In total, 15 papers were selected (Table 1).
In Figure 1 we reported the PRISMA statement, while the RQS v.2 was reported in Table 2.
Of the 15 included papers, 13 were qualified as conventional radiomics approaches, and two were as Deep Learning radiomics [21,22].
Within the conventional radiomics group, semi-automated segmentation was the most common approach to ROI identification (10 studies) followed by manual delineation (three). Multiple ROI identification with the assessment of feature robustness to inter-observer variability was carried out in four studies out of 15. The total number of features initially extracted from each ROI ranged from six [23] to 3,488 [26]; the most common features were PET semi-quantitative parameters, first-order statistics and texture features. Feature selection was performed in the majority of the studies (12), either through one single method or a combination of them. The Least Absolute Shrinkage and Selection Operator (LASSO) was the most popular approach for this task (nine studies), followed by redundancy analysis via Pearson’s correlation coefficient (five). Multivariate logistic regression was the preferred model for endpoint prediction (six studies); other approaches were weighted linear models (three) and Random Forest classification (two).
The two papers investigating Deep Learning were based, respectively, on a two-dimensional Small Residual Convolutional Network (SresCNN [24]) and a three-dimensional CNN [22]. Specifically, in [24] the input to the network was a series of planar boxes from consecutive slices clipped around the ROI, whereas in [22] it was a cube-shape volume around the inspected lesion. Data augmentation was used in both papers.

3.2. Baseline PET for the Prediction of Biomarker Expression

PD-L1 expression status and tumor mutational burden are considered both valuable predictive factors in NSCLC patients’ candidates to ICIs. Nevertheless, a variable number of patients cannot benefit from ICIs, independently of PD-L1 score, through a mechanism not yet fully known [25], thus requiring the identification of additional predictive parameters. In this context, the rising role of radiomics could contribute to identifying novel biomarkers useful for the correct identification of patients who will most likely benefit from ICIs.
Several papers have investigated the potential role of radiomics to predict PD-L1 expression status in NSCLC patients before starting ICI treatment [21,26,27,28]. Jian et al. [26] extracted radiomics features from [18F]FDG PET/CT of 399 NSCLC patients. After reduction with the LASSO algorithm, three predictive models were developed based on features from CT alone, PET alone, and PET/CT combined images. For PD-L1 evaluated by using the SP142 kit, the AUCs for predicting PD-L1 >1% vs. PD-L1>50% were 0.97 vs. 0.80, 0.61 vs. 0.65, and 0.97 vs. 0.77, respectively for CT alone, PET alone, and PET/CT combined images. On the other hand, for PD-L1 evaluated with 28-8 kit, AUC was 0.86, 0.62, and 0.85 for predicting PD-L1 >1%, and 0.91, 0.75, and 0.88 for predicting PD-L1 >50%, respectively for CT alone, PET alone, and PET/CT combined images.
Li et al. [27] proposed a combined model between radiomics features extracted from [18F]-FDG PET/CT, and clinicopathologic variables (i.e., age, gender, tumor location, histology type and grade, carcinoembryonic antigen level, smoking history, Ki-67. Overall, 255 patients were enrolled and divided into training (n=170) and validation groups (n=85). Eighteen out of 80 radiomics features (six from CT and twelve from PET) were useful to predict PD-L1 >1%, and 7 (four from CT and three from PET) for PD-L1 >50%. The combined model for the prediction of PD-L1 >1% showed an AUC score of 0.757 (95% CI: 0.699-0.808), whereas for the prediction of PD-L1 >50% AUC was 0.814 (95% CI: 0.761-0.860).
Similarly, Zhao et al. [28] built and validated a radiomics model, a clinical model, and their combination for predicting PDL-1 expression status (if >= 1%) in NSCLC patients. After the LASSO algorithm and 10-fold cross-validation, two optimal radiomics features (Gray Level Run Length Matrix (GLRLM)_Run percentage (RP) and Shape_Sphericity) were selected. The AUC values of the combined model were significantly higher than those of the clinical model both in the training (0.718 vs 0.638, p=0.004) and validation group (0.769 vs 0.640, p=0.007), while there were no significant differences between the combined and radiomics models in both training and validation cohort. Hence, based on the combined model, an individualized nomogram was developed, showing good consistency between the predictive probability and the actual predicted probability in the training group (χ2=1.463, p=0.481) and the validation group (χ2=1.563 p=0.458) with no significant differences between different PET/CT scanners.
Moreover, in a recent study, Mu et al. [21] investigated the potential role of a deeply learned score (DLS) for predicting PD-L1 expression status in 697 NSCLC patients. The results were satisfying, with an AUC ≥ 0.82 for discriminating PD-L1 positive (if >= 1%) vs negative patients. Interestingly, DLS paired immunohistochemistry derived PD-L1 status for predicting progression-free survival (PFS) and OS.
The characterization of the tumor microenvironment (TME), including tumor-infiltrating lymphocytes (TILs) CD3+ and CD8+, represents another independent biomarker, even though the heterogeneity between primary tumor and metastatic lesions, as well as the difficulty of biopsy in some patients, makes its use unsatisfactory for monitoring ICIs efficacy [29,30]. Two studies have explored the potential role of radiomics to predict the composition of TME [22,31]. Park et al. [22] developed a deep-learning model to estimate the TME in lung adenocarcinoma using data from [18F]FDG PET/CT and RNA sequencing of 93 patients. The cytolytic activity score (CytAct) was used as an indirect biomarker for TME, as it represents CD8+ T cell activity and it is easy to calculate. The model was validated in two independent cohorts (n=43, n=16, respectively) and showed a positive correlation with CytAct of RNA sequencing from both (rho = 0.32 and 0.47, respectively). On the other hand, in the ICI cohort, predicted CytAct was inversely correlated with tumor size after ICI treatment (rho = -0.54). In addition, higher minimum predicted CytAct was also associated with prolonged PFS and OS (HR 0.25, p=0.001 and HR 0.18, p=0.004, respectively).
More recently, Tong et al. [31] applied a machine learning model to evaluate the TME phenotype combining [18F]FDG PET/CT data and clinical characteristics of NSCLC patients from Daping Hospital (DPH) and The Cancer Imaging Archive (TCIA). First, the Delong test demonstrated that PET/CT model outperformed the CT alone model to predict the CD8 expression. Later, PET/CT radiomics-clinical model, integrating significant clinical features with Rad-score, was able to predict TME status in NSCLC (training AUC of 0.932, testing AUC of 0.920), showing better performance compared to the clinical and radiomics models, separately (AUC= 0.932 vs. 0.868 vs. 0.907, respectively). In addition, the radiomics-clinical combined model was also applied in the TCIA cohort for predicting the TME phenotype. Based on the combined model, patients were classified into predicted CD8-high and CD8-low groups. The first group showed significantly higher immune scores and more activated immune pathways than the second group, implicating better response when treated with ICI.

3.3. The Prediction of Response to Immunotherapy

Radiomics could contribute both in predicting the response to immunotherapy and to identify therapy-responsive patients accurately and early: from the clinical point of view, each additional month of ineffective therapy can be crucial for metastatic NSCLC patients and costly for public health.
So far, the radiomics potentialities from baseline PET as predictive parameters were tested by two authors [21,32]. Mu et al. [21] published one of the first studies investigating the predictive role of radiomics in NSCLC patients treated with ICI. The authors extracted radiomics features from baseline CT, PET, and PET/CT-fused images and found that characteristics related to heterogeneity (i.e., short run low gray emphasis or short zone emphasis) could reliably predict durable benefit from ICI treatment (AUC of 0.86 for training, 0.83 for retrospective, and 0.81 for prospective test cohorts). Nevertheless, the main limitation of this study was the lack of PD-L1 expression data for many enrolled patients, hindering a direct comparison of the authors’ model with the PD-L1 status. Polverari et al. [[32]] found that NSCLC patients with elevated TLG, volume, and high tumor heterogeneity in asymmetry (i.e., skewness) and kurtosis were more likely to incur into disease progression during ICI treatment, although the lack of a robust validation cohort represents the main limitation of this study.
Radiomics features from serial PET/CT scans before, during or after ICIs were assessed by three authors [23,33,34]. In 2020, Valentinuzzi et al. [23] created a [18F]FDG PET radiomics signature (iRADIOMICS), consisting of the most promising radiomics features extracted by the [18F]-FDG images, able to predict the response of metastatic NSCLC (stage IV) to pembrolizumab compared to the clinical standards (PD-L1 immunohistochemistry and iRECIST). Thirty patients receiving pembrolizumab were scanned with [18F]FDG PET/CT at baseline, months 1 and 4. Response to therapy was defined as OS > 14.9 months. iRADIOMICS (baseline), iRECIST (months 1 and 4), and PD-L1 (baseline) signatures were constructed using univariate or multivariate logistic regression analyses. At baseline PET, none of the standard volume-based features (volume, SUVmax) were able to discriminate responders from non-responders. On the contrary, the predictive power of the baseline-iRADIOMICS signature was higher than the PD-L1 signature (AUC of 0.81 vs 0.60) and comparable to month 1 and month 4 iRECIST signatures (AUC of 0.79 and 0.81, respectively), allowing earlier identification of the response by at least one month. To further validate the predictive ability of all models, the accuracy of predictions was calculated using 5-fold cross-validation. Multivariate baseline iRADIOMICS was found to be superior to the current standards (PD-L1 and iRECIST signatures). Both conventional parameters and radiomics features extracted by month 1 and month 4 PET/CT images were not significantly different between responders and non-responders, with the only exception of the volume of the lesion, after 1 month from the start of ICIs (p = 0.035, AUC = 0.75 (0.55-0.95)). Despite this promising data, authors analysed only primary tumors, yet neglected lymph nodes (LN) and distant metastases (DM).
Tankyevych et al. [33] retrospectively evaluated 83 patients with locally advanced or metastatic NSCLC treated with immunotherapy. They aimed to assess the ability of radiomics features from baseline (PET/CT0) and both early (PET/CT1 = 6-8 weeks after the initiation of therapy) and late (PET/CT2 = 3 months after the initiation of therapy) follow-up [18F]FDG PET/CT scans, as well as their evolution (delta-radiomics), to predict durable clinical benefit (DCB), progression (according to PERCIST at the first restaging, iPERCIST and RECIST1.1 after 3 months of treatment), response to therapy, PFS and OS. Seven multivariate models with different combinations of clinical and radiomics parameters (CP, PET, CT, PET-CP, CT-CP, PET-CT-CP) were trained on a subset of patients (75%) using LASSO, and random forest classification with 10-fold cross-validation to predict the outcome. Model validation was performed on the remaining patients (25%). At baseline (PET/CT0), standard PET-based features (SUVs, MTV, TLG) were not able to significantly discriminate between patients in terms of progression, DCB or survival; differently, several radiomics and delta-radiomics parameters predicted the outcome with better performance than clinical and conventional PET parameters (AUC > 0.8), with slightly better performance of parameters extracted from baseline (PET/CT0) and at month 2 (PET/CT1) PET/CT than delta-radiomics parameters. Overall, PET and CT parameters extracted from PET/CT1 were greater predictors than those at baseline. Furthermore, several multivariate models performed well, especially with radiomics data extracted from PET/CT0 imaging, for both progression prediction (AUC of 1 and 0.96) and DCB (AUC of 0.85 and 0.83 with the PET-CT-CP model).
Recently, Cui et al. [34] performed a prospective study on 30 stages III NSCLC patients (without brain metastasis according to MRI) who received [18F]FDG PET/CT baseline (13 patients) and preoperative scans (29 patients) three weeks after the completion of neoadjuvant treatment (toripalimab + chemotherapy). Lung lesions were delineated by three different nuclear medicine physicians. Six conventional PET parameters, 102 radiomics features (using the Python package Pyradiomics), and delta features were included in the analysis. The radiological and metabolic response, in terms of complete pathological response (CPR), were assessed by iRECIST and iPERCIST, respectively; the major pathological response (MPR) was evaluated in the surgical specimen. Twenty patients achieved MPR and 16 of them achieved CPR. For delta PET features, five SUV statistics features (SUVmax, SUVpeak, SULmax, SULpeak, and TLG) and one radiomics feature (Delta-original-GLDMDependenceNonUniformity [Delta-GLDM-DN]) distributions significantly differed both in CPR and non-CPR, MPR and non-MPR subgroups. No significant correlation, neither between the radiological and the pathological response, nor among PD-L1, driver gene status, and baseline PET features were found. For preoperative PET features, in univariate analysis, five SUV statistics features and two radiomics features were significantly associated with pathological responses. In multivariate analysis, SUVmax, SUVpeak, SULpeak, and End-GLDM-LDHGLE were independently associated with CPR. SUVpeak and SULpeak performed better than SUVmax and SULmax for MPR prediction. Despite the absence of external validation and the scarce cohort, this study appears remarkable for the homogeneous nature of the evaluated sample and the comprehensive number of PET and clinical data evaluated. Again, the data from baseline PET resulted less informative than those obtained from preoperative PET (post-immunotherapy PET), which might provide additional valuable information on the TME and heterogeneity to differentiate residual tumor cells and influential immune cells [35].

3.4. The Prediction of Adverse Events Correlated with Immunotherapy by [18F]FDG PET/CT and Radiomics

Mu and colleagues [36] evaluated the role of radiomics analysis in predicting the occurrence of irAEs in 146 pts with histologically confirmed advanced stage (IIIB and IV) NSCLC who underwent [18F]FDG PET/CT 6 months before treatment initiation. Radiomics features extracted from baseline PET, CT, and PET/CT fusion images were used to generate a radiomics score (RS) to quantify the patient risk for developing irSAE by an improved LASSO method. A weighted multivariable logistic regression analysis was then used to develop a nomogram model to predict irSAE, which was assessed by its calibration, discrimination, and clinical usefulness. The authors found that the radiomics nomogram, incorporating the RS, type of immune checkpoint blockade, and dosing schedule, was able to predict irSAE with an AUC of 0.92 (CI: 0.86, 0.98), 0.92 (CI: 0.86, 0.99), and 0.88 (CI: 0.78, 0.97) in the training, test, and prospective validation cohorts, respectively.
In another study [24], the same group applied radiomics analysis on [18F]FDG PET/CT images in 175 patients with NSCLC (stage IIIB and IV) to specifically predict the risk of cachexia, a syndrome that induces progressive functional impairment accounting for 20% of cancer-related deaths [[37]] and promotes primary resistance to ICI. The authors evaluated the durable clinical benefit (DCB, PFS > 6 months) following ICI considering PFS and OS as main endpoints. As a result, the RS was significantly different between cachexic and non-cachexic pts in the training cohort (p < 0.001), which was validated in the test cohort (p = 0.003) and external test cohort (p = 0.04). The RS predicted the risk of cachexia with AUCs ≥ 0.74 in the training, test, and external test cohorts. PFS and OS were significantly shorter among patients with higher radiomics-based cachexia probability in all three cohorts. Furthermore, the RS identified patients with DCB reaching AUCs ≥ 0.66 in all three cohorts. In addition, the authors observed that body mass index (BMI), Eastern Clinical Oncology Group (ECOG), distant metastasis and RS were significant and independent predictors of cachexia.

4. Discussion

4.1. Clinical Assessment

As emerged from the present systematic review, few data are now available about the utility of radiomics in NSCLC candidates or undergoing immunotherapy. Indeed, only 15 papers were selected and the content for each of them was highly variable.
Some papers discussed the utility of radiomics or deep learning analysis to predict the expression of PDL-1 or to evaluate the TME, with controversial results. In fact, only one study performed by using deep learning analysis demonstrated an increase in the AUC for the prediction of PDL-1 expression, as compared to the standard current criteria. Moreover, only two papers aimed to predict the response to immunotherapy by using radiomics. Finally, some efforts have been made for understanding if radiomics can overpass some criticisms in the evaluation of response to immunotherapy. Again, few data are now available and in two reports, the authors agreed that rather than baseline PET/CT, radiomics analysis from the first scan after the start of immunotherapy or before surgery can be helpful in predicting the success of the therapy.

4.2. Radiomics Evaluation

Previous works suggested that Deep Learning may achieve better performance than conventional radiomics at the cost, however, of interpretability issues [38,39]. Notably, most of the studies considered in this review (13 out of 15) were based on conventional radiomics and only two on Deep Learning. This is partly the consequence of Deep Learning being relatively newer and more complex than conventional radiomics. In addition, whereas significant work has been done towards the standardization of image biomarkers in conventional radiomics [40], we cannot affirm the same for Deep Learning. We should also consider that conventional radiomics is possibly more affordable than Deep Learning in terms of computational resources and data availability and that the former can rely on user-friendly standalone packages (e.g., LIFEx) that do not require high-end coding skills.
One critical point of all the selected studies is that none of them attempted at comparing the radiomics vs. Deep Learning in terms of prediction accuracy, interpretability and/or robustness. Likewise, no studies addressed the possibility of combining the two approaches. Finally, the lack of data sharing including images, clinical meta-data and/or code/algorithms is a limitation that should be addressed in future research.
Independently of their main methodology, all the included studies in this review achieved a low RQS: indeed, none scored above 50% and most of them averaged around one-third of the available points. The “top-tier” papers were all published by the same group [24,36,41,42], which developed and applied a method to investigate various outcomes of NSCLC patients treated with immunotherapy. The quality issue of radiomics research is well-known and has been reported in past reviews [43,44]. Namely, the scores reflect the way RQS is structured, rewarding mainly prospective studies, as well as the presence of multiple training and validation cohorts. Moreover, many of the RQS checkpoints are tied to a series of procedures related to textural data harmonization and statistical optimization, which have been sparsely implemented by the considered studies. Finally, one key point is the level of automation provided by the presented radiomics technique: to obtain the highest score the method should be able to provide a prediction for the chosen outcome reliably and without human intervention [45]. The presented radiomics analyses are still far from such a lofty goal, even though some of them could be further developed past their current automation level. In general, the requirements of the RQS are stringent; moreover, the second version of the test bench added further requirements and statistical fine-tuning to the score composition. Indeed, striving for data harmonization, statistical integrity, and external validation could be the only way to progress radiomics. In our opinion, three key points should be addressed, when developing radiomics research studies. First, multicentre prospective studies should be preferred since they ensure data homogeneity and foster higher external reproducibility of the studies. In this setting, long-term thinking and allocation of resources should be favored: large, well-planned prospective studies with multiple external validations could provide the most valuable data. The second version of RQS added the international nature of a multicentre study as a further checkpoint, which is a very important issue since the characteristics of the subjects can vary significantly across populations. The scoring is in fact affected by the study design (whether it is single-, multi- or international multi-centre), the training dataset coming from different centres and the existence of prospective – ideally multi-centre – validation arms. Secondly, the key radiomics characteristics in predicting a specific outcome should be identified, and their relationship with their biological counterparts investigated ([46]). Radiomics “signatures”, while producing an easy-to-read scoring system, offer no insight into the reasons that cause radiomics to be effective in a particular clinical task. Moreover, their results can be hard to reproduce [47,48]. Identifying the key variables (such as entropy, homogeneity, and second-order parameters), could allow linking these characteristics to specific clinical features (e.g., vascularization, tumor response) and even predict the effectiveness of target therapy in specific cases [49]. Finally, in future, the identified key radiomics features could be parametrized and shown to the clinician as a functional visual map, which could be overlaid on the standard imaging [1]. Such a tool would be pivotal to ensure the transition of radiomics to the clinical setting, with multidisciplinary tools that could be read by all actors participating in the patients’ treatment process, such as radiologists, surgeons, radiation oncologists, nuclear medicine physicians, and medical physicists. Such a path could replicate the success of functional imaging achieved by PET in the latest decades.

5. Conclusions

In conclusion, radiomics is promising in the evaluation of TME and for the prediction of response to immunotherapy, but some limitations should be overpassed. First of all, the study design should be made by specific criteria or methodology. Second, prospective studies are required in order to overcome heterogeneity. Finally, the inclusion in clinical practice of a simple tool able to adequately analyse the image appeared mandatory for its larger use. Prospective well designed studies, by including a large population, are mandatory.

Author Contributions

Conceptualization, L.E. and L.F.; methodology, L.E.; software, F.F., F.B., L.M.; investigation, all; resources, all; data curation, all; writing—original draft preparation, all; writing—review and editing, B.P., L.E. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Global Burden of Disease Cancer Collaboration; Fitzmaurice Ch, Abate D, Abbasi N; et al. Global, Regional, and National Cancer Incidence, Mortality, Years of Life Lost, Years Lived with Disability, and Disability-Adjusted Life-Years for 29 Cancer Groups, 1990 to 2017: A Systematic Analysis for the Global Burden of Disease Study. JAMA Oncol 2019, 5, 1749–1768. [CrossRef]
  2. Revels SL, Lee JM. Anti-angiogenic therapy in nonsquamous non-small cell lung cancer (NSCLC) with tyrosine kinase inhibition (TKI) that targets the VEGF receptor (VEGFR): Perspective on phase III clinical trials. J Thorac Dis. 2018, 10, 617–20. [CrossRef] [PubMed]
  3. Guo Q, Liu L, Chen Z, Fan Y, Zhou Y, Yuan Z; et al. Current treatments for non-small cell lung cancer. Front Oncol. 2022;12. [CrossRef]
  4. Mithoowani H, Febbraro M. Non-Small-Cell Lung Cancer in 2022: A Review for General Practitioners in Oncology. Curr Oncol. 2022, 29, 1828–1839. [CrossRef]
  5. Brahmer J, Reckamp KL, Baas P, Crinò L, Eberhardt WEE, Poddubskaya E; et al. Nivolumab versus Docetaxel in Advanced Squamous-Cell Non–Small-Cell Lung Cancer. N Engl J Med. 2015, 373, 123–35. [CrossRef]
  6. Onoi K, Chihara Y, Uchino J, Shimamoto T, Morimoto Y, Iwasaku M; et al. Immune Checkpoint Inhibitors for Lung Cancer Treatment: A Review. J Clin Med. 2020, 9, 1362. [CrossRef]
  7. Melosky B, Cheema PK, Brade A, McLeod D, Liu G, Price PW; et al. Prolonging Survival: The Role of Immune Checkpoint Inhibitors in the Treatment of Extensive-Stage Small Cell Lung Cancer. Oncologist 2020, 25, 981–992. [CrossRef]
  8. Blons H, Garinet S, Laurent-Puig P, Oudart J-B. Molecular markers and prediction of response to immunotherapy in non-small cell lung cancer, an update. J Thorac Dis. 2019, 11, S25–36. [CrossRef]
  9. Xu C, Chen Y-P, Du X-J, Liu J-Q, Huang C-L, Chen L; et al. Comparative safety of immune checkpoint inhibitors in cancer: Systematic review and network meta-analysis. BMJ. 2018;k4226. [CrossRef]
  10. Rowe SP, Pomper MG. Molecular imaging in oncology: Current impact and future directions. CA Cancer J Clin. 2022, 72, 333–52. [CrossRef]
  11. Evangelista L, Cuppari L, Menis J, Bonanno L, Reccia P, Frega S; et al. 18F-FDG PET/CT in non-small-cell lung cancer patients. Nucl Med Commun. 2019, 40, 802–807. [CrossRef]
  12. Kudura K, Ritz N, Kutzker T, Hoffmann MHK, Templeton AJ, Foerster R; et al. Predictive Value of Baseline FDG-PET/CT for the Durable Response to Immune Checkpoint Inhibition in NSCLC Patients Using the Morphological and Metabolic Features of Primary Tumors. Cancers 2022, 14, 6095. [CrossRef]
  13. Bianconi F, Palumbo I, Fravolini ML, Rondini M, Minestrini M, Pascoletti G; et al. Form Factors as Potential Imaging Biomarkers to Differentiate Benign vs. Malignant Lung Lesions on CT Scans. Sensors 2022, 22, 5044. [CrossRef]
  14. Aide N, Weyts K, Lasnon C. Prediction of the Presence of Targetable Molecular Alteration(s) with Clinico-Metabolic 18 F-FDG PET Radiomics in Non-Asian Lung Adenocarcinoma Patients. Diagnostics 2022, 12, 2448. [CrossRef]
  15. Zhang Y, Hu H-H, Zhou S-H, Xia W-Y, Zhang Y, Zhang J-P; et al. PET-based radiomics visualizes tumor-infiltrating CD8 T cell exhaustion to optimize radiotherapy/immunotherapy combination in mouse models of lung cancer. Biomark Res. 2023, 11, 10. [CrossRef]
  16. Castello A, Castellani M, Florimonte L, Urso L, Mansi L, Lopci E. The Role of Radiomics in the Era of Immune Checkpoint Inhibitors: A New Protagonist in the Jungle of Response Criteria. J Clin Med. 2022, 11, 1740. [CrossRef]
  17. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ 2009, 339, b2535. [CrossRef]
  18. Hosny A, Aerts HJ, Mak RH. Handcrafted versus deep learning radiomics for prediction of cancer therapy response. Lancet Digit Heal. 2019, 1, e106–e107. [CrossRef]
  19. Bianconi F, Palumbo I, Spanu A, Nuvoli S, Fravolini ML, Palumbo B. PET/CT Radiomics in Lung Cancer: An Overview. Appl Sci. 2020, 10, 1718. [CrossRef]
  20. Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong EEC, van Timmeren J; et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol. 2017, 14, 749–762. [CrossRef]
  21. Mu W, Jiang L, Shi Y, Tunali I, Gray JE, Katsoulakis E; et al. Non-invasive measurement of PD-L1 status and prediction of immunotherapy response using deep learning of PET/CT images. J Immunother cancer. 2021;9. [CrossRef]
  22. Park C, Na KJ, Choi H, Ock C-Y, Ha S, Kim M; et al. Tumor immune profiles noninvasively estimated by FDG PET with deep learning correlate with immunotherapy response in lung adenocarcinoma. Theranostics 2020, 10, 10838–10848. [CrossRef]
  23. Valentinuzzi D, Vrankar M, Boc N, Ahac V, Zupancic Z, Unk M; et al. [18F]FDG PET immunotherapy radiomics signature (iRADIOMICS) predicts response of non-small-cell lung cancer patients treated with pembrolizumab. Radiol Oncol. 2020, 54, 285–94. [CrossRef]
  24. Mu W, Katsoulakis E, Whelan CJ, Gage KL, Schabath MB, Gillies RJ. Radiomics predicts risk of cachexia in advanced NSCLC patients treated with immune checkpoint inhibitors. Br J Cancer 2021, 125, 229–239. [CrossRef]
  25. Doroshow DB, Bhalla S, Beasley MB, Sholl LM, Kerr KM, Gnjatic S; et al. PD-L1 as a biomarker of response to immune-checkpoint inhibitors. Nat Rev Clin Oncol. 2021, 18, 345–62.
  26. Jiang M, Sun D, Guo Y, Guo Y, Xiao J, Wang L; et al. Assessing PD-L1 Expression Level by Radiomic Features From PET/CT in Nonsmall Cell Lung Cancer Patients: An Initial Result. Acad Radiol. 2020, 27, 171–179. [CrossRef]
  27. Li J, Ge S, Sang S, Hu C, Deng S. Evaluation of PD-L1 Expression Level in Patients With Non-Small Cell Lung Cancer by 18F-FDG PET/CT Radiomics and Clinicopathological Characteristics. Front Oncol. 2021;11. [CrossRef]
  28. Zhao X, Zhao Y, Zhang J, Zhang Z, Liu L, Zhao X. Predicting PD-L1 expression status in patients with non-small cell lung cancer using [18F]FDG PET/CT radiomics. EJNMMI Res. 2023, 13, 4. [CrossRef]
  29. Kim H, Kwon HJ, Han YB, Park SY, Kim ES, Kim SH; et al. Increased CD3+ T cells with a low FOXP3+/CD8+ T cell ratio can predict anti-PD-1 therapeutic response in non-small cell lung cancer patients. Mod Pathol. 2019, 32, 367–375. [CrossRef]
  30. Geng Y, Shao Y, He W, Hu W, Xu Y, Chen J; et al. Prognostic Role of Tumor-Infiltrating Lymphocytes in Lung Cancer: A Meta-Analysis. Cell Physiol Biochem. 2015, 37, 1560–1571. [CrossRef]
  31. Tong H, Sun J, Fang J, Zhang M, Liu H, Xia R; et al. A Machine Learning Model Based on PET/CT Radiomics and Clinical Characteristics Predicts Tumor Immune Profiles in Non-Small Cell Lung Cancer: A Retrospective Multicohort Study. Front Immunol. 2022, 13, 859323. [CrossRef]
  32. Polverari G, Ceci F, Bertaglia V, Reale ML, Rampado O, Gallio E; et al. 18F-FDG Pet Parameters and Radiomics Features Analysis in Advanced Nsclc Treated with Immunotherapy as Predictors of Therapy Response and Survival. Cancers 2020;12. [CrossRef]
  33. Tankyevych O, Trousset F, Latappy C, Berraho M, Dutilh J, Tasu JP; et al. Development of Radiomic-Based Model to Predict Clinical Outcomes in Non-Small Cell Lung Cancer Patients Treated with Immunotherapy. Cancers 2022;14. [CrossRef]
  34. Cui Y, Lin Y, Zhao Z, Long H, Zheng L, Lin X. Comprehensive 18F-FDG PET-based radiomics in elevating the pathological response to neoadjuvant immunochemotherapy for resectable stage III non-small-cell lung cancer: A pilot study. Front Immunol. 2022, 13, 994917. [CrossRef]
  35. Wang D, Zhang X, Liu H, Qiu B, Liu S, Zheng C; et al. Assessing dynamic metabolic heterogeneity in non-small cell lung cancer patients via ultra-high sensitivity total-body [18F]FDG PET/CT imaging: Quantitative analysis of [18F]FDG uptake in primary tumors and metastatic lymph nodes. Eur J Nucl Med Mol Imaging. 2022, 49, 4692–4704. [CrossRef]
  36. Mu W, Tunali I, Qi J, Schabath MB, Gillies RJ. Radiomics of 18 F Fluorodeoxyglucose PET/CT Images Predicts Severe Immune-related Adverse Events in Patients with NSCLC. Radiol Artif Intell. 2020, 2, e190063. [CrossRef]
  37. Tisdale, MJ. Cachexia in cancer patients. Nat Rev Cancer. 2002, 2, 862–871. [Google Scholar] [CrossRef] [PubMed]
  38. Papadimitroulas P, Brocki L, Christopher Chung N, Marchadour W, Vermet F, Gaubert L; et al. Artificial intelligence: Deep learning in oncological radiomics and challenges of interpretability and data harmonization. Phys Medica. 2021, 83, 108–121. [CrossRef]
  39. Astaraki M, Yang G, Zakko Y, Toma-Dasu I, Smedby Ö, Wang C. A Comparative Study of Radiomics and Deep-Learning Based Methods for Pulmonary Nodule Malignancy Prediction in Low Dose CT Images. Front Oncol. 2021;11. [CrossRef]
  40. Zwanenburg A, Vallières M, Abdalah MA, Aerts HJWL, Andrearczyk V, Apte A; et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 2020, 295, 328–338. [CrossRef]
  41. Mu W, Tunali I, Gray JE, Qi J, Schabath MB, Gillies RJ. Radiomics of 18F-FDG PET/CT images predicts clinical benefit of advanced NSCLC patients to checkpoint blockade immunotherapy. Eur J Nucl Med Mol Imaging 2020, 47, 1168–1182. [CrossRef]
  42. Fiz F, Viganò L, Gennaro N, Costa G, La Bella L, Boichuk A; et al. Radiomics of Liver Metastases: A Systematic Review. Cancers (Basel). 2020;12. [CrossRef]
  43. Dercle L, McGale J, Sun S, Marabelle A, Yeh R, Deutsch E; et al. Artificial intelligence and radiomics: Fundamentals, applications, and challenges in immunotherapy. J Immunother cancer. 2022;10. [CrossRef]
  44. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B. Radiomics in medical imaging-”how-to” guide and critical reflection. Insights Imaging. 2020, 11, 91. [CrossRef]
  45. Laudicella R, Comelli A, Liberini V, Vento A, Stefano A, Spataro A; et al. [68Ga]DOTATOC PET/CT Radiomics to Predict the Response in GEP-NETs Undergoing [177Lu]DOTATOC PRRT: The “Theragnomics” Concept. Cancers (Basel). 2022;14. [CrossRef]
  46. Binczyk F, Prazuch W, Bozek P, Polanska J. Radiomics and artificial intelligence in lung cancer screening. Transl lung cancer Res. 2021, 10, 1186–1199. [CrossRef]
  47. Zhao, B. Understanding Sources of Variation to Improve the Reproducibility of Radiomics. Front Oncol. 2021, 11, 633176. [Google Scholar] [CrossRef]
  48. Rao S-X, Lambregts DM, Schnerr RS, Beckers RC, Maas M, Albarello F; et al. CT texture analysis in colorectal liver metastases: A better way than size and volume measurements to assess response to chemotherapy? United Eur Gastroenterol J. 2016, 4, 257–263. [CrossRef]
  49. Zhou J, Zou S, Kuang D, Yan J, Zhao J, Zhu X. A Novel Approach Using FDG-PET/CT-Based Radiomics to Assess Tumor Immune Phenotypes in Patients with Non-Small Cell Lung Cancer. Front Oncol. 2021, 11, 769272. [CrossRef]
Preprints 73234 g001
Figure 1. PRISMA statement for the selection of the papers.
Figure 1. PRISMA statement for the selection of the papers.
Preprints 73234 g001
Table 1. Main characteristics of the selected studies.
Table 1. Main characteristics of the selected studies.
Author, ref Year of pub. Design Sample size Histology Type of ICIs Histopathology correlation Software Model External validation cohort Outcome measures Relevant radiomics indexes RQS
Jiang et al. [26] 2019 R 399 NSCLC (SCC and Adenocarcinoma) Atezolizumab and Nivolumab Yes ITK V. 3.6.1 Logistic regression and random forest Na PD-L1 expression Shape, IQR, GLCM_JointAverage, median, NGTDM_contrast 22 (33,3%)
Polverari et al. [32] 2020 R 57 mixed histologies Mixed Yes LifeX Univariate analysis Na PD-L1 expression; progression status Coarseness, GLZLM_ZLNU, Kurtosis, Skewness, GLZLM_LZE, GLRLM_RP/SRE/HGRE, GLCM_Homogeneity 13 (19,7%)
Mu et al.
[36]		 2020 R/P 146 (R), 48 (P) NSCLC (123 ADC and 71 SCC) N/S Yes In-house software Logistic regression and Cox multivariate regression Na Durable clinical benefit, PFS, and OS P/R radiomics signatures 28 (42,4%)
Mu et al. [41] 2020 R/P 146 (R), 48 (P) NSCLC (123 ADC and 71 SCC) Multiple Na In-house software Multivariable regression analysis Na Immune-related adverse events Radiomic signature (KLD_SZLGE and KLD_SRLGE) 26 (39,39%)
Park et al. [22] 2020 R 29 NSCLC (ADC) Pembrolizumab (10)
Nivolumab (18)
Atezolizumab (1)		 Yes LifeX v 4 Deep Learning Yes Cytolitic activity; tumour response, PFS, and OS N/S 16 (26,23%)*
Valentinuzzi et al. [23] 2020 P 30 NSCLC (17 ADC, 8 SCC, and 5 other) Pembrolizumab Na In-house software Univariate analysis and Cox regression model Na OS GLRLM_SRE 22 (33,3%)
Li et al. [27] 2021 R 255 NSCLC (SCC and Adenocarcinoma) N/S Yes LifeX v 7 Logistic regression Na PD-L1 expression (>1% and >50%) N/S (12 and 3 feature for >1% and >50%, respectively) 20 (30,3%)
Mu et al. [24] 2021 R 210 NSCLC (109 ADC and 66 SCC) N/S (anti PD-1 and anti PD-L1) N MatLab 2020.a Uni/multivariable regression analysis Y Caxhexia; Durable clinical benefit, PFS, and OS Radiomic signature (SRHGE and LZLGE) 26 (39,39%)
Mu et al. [21] 2021 R/P 648 (R), 49 (P) NSCLC (531 ADC and 166 SCC) N/S Y ITK Small residual convolutional network (SResCNN) Y PD-L1 expression; Durable clinical benefit, PFS, and OS N/S 26 (42,6%)
Zhou et al. [49] 2021 R 103 28 SCC and 75 other N/S Y LifeX v 5.1 Univariate analysis and logistic regression N PD-L1 and CD8 expression GLRLM_LRHGE, GLZLM_SZE, SUVmax, NGLDM_Contrast 23 (34,85%)
Tankyevych et al. [33] 2022 R 83 mixed histologies Mixed Y PyRadiomics Multivariate model N Survival, Progression, durable clinical benefit Skewness, median, NGTDM_Complexity, GLCM_Autocorrelation and GLCM_imc1 25 (37,9%)
Tong et al. [31] 2022 R 221 NSCLC (N/S) N/S Y ITK V. 3.8 Clinical-radiomics models; machine learning N CD-8 expression GLCM_IMC1, GLSZM_SZLGE, GLTDM_LGE, Histogram Energy, GLTDM_Entropy 24 (36,36%)
Cui et al. [34] 2022 P 29 NSCLC (mixed histologies) Toripalimab Y Pyradiomics Logistic regression N Pathological response of the primary Delta SUV-indices; EOT SUV indices; EOT MTV/TLG, EOT uniformity and EOT GLDM_LDHGLE 21 (31,82%)
Wang et al. [35] 2022 P 30 NSCLC (16 ADC, 12 SCC, and 2 other) None** Y N/S Univariate analysis Y Heterogeneity, immune infiltrate Entropy 16 (24,24%)
Zhao et al. [28] 2023 R 334 NSCLC (163 ADC, 59 SCC, and 112 other) Pembrolizumab Y LifeX v 7 Univariate analysis and logistic regression N PD-L1 expression GLRLM_RP 20 (30,30%)
* the deep-learning specific scoring was used; **the correlation with the immune infiltrate suggests that dynamic analysis might be used to evaluate treatment with ICI; ADC: adenocarcinoma; CD-8: cluster of differentiation 8; EOT: end of treatment; GLCM: gray-level co-occurrence matrix; GLDM_LDHGLE: gray level dependence matrix_ large dependence high gray level emphasis; GLRLM_RP/SRE/HGRE: grey-level run length matrix_run percentage/short-run emphasis/ high gray-level run emphasis; GLSZM: gray level size zone matrix; GLTDM: gray level total displacement matrix; GLZLM: gray-level zone length matrix; IMC1: informational measure of correlation 1; IQR: interquartile range; KLD_SZLGE/SRLGE: Kullback-Leibler divergence_ short-zone gray-level emphasis/ short-run low gray-level emphasis; LGE: low gray-level emphasis; LRHGE: long-run high gray-level emphasis; LZE: long-zone emphasis; LZLGE: long-zone low gray-level emphasis; MTV: metabolic tumor volume; N: no; NGTDM: neighborhood grey tone difference matrix; N/S: not specified ; NSCLC: non-small cell lung carcinoma; OS: overall survival; P: prospective; PD-L1: programmed death-ligand 1; PFS: progression free survival; R: retrospective; RQS: radiomic quality score ; SCC: squamous cell carcinoma; SRHGE: short run high gray-level emphasis; SUVmax: maximum standardized uptake value; SZE: short-zone emphasis; SZLGE: short-zone low gray-level emphasis; TLG: total lesion glycolysis; Y: yes; ZLNU: zone length nonuniformity.
Table 2. Radiomics quality score (v 2.0) of the included studies.
Table 2. Radiomics quality score (v 2.0) of the included studies.
Authors (PMID) Rater
FB FF LM Consensus
Jiang et al. [26] 22 22 22 22
Polverari et al. [32] 13 13 15 13
Mu et al. [36] 23 26 26 28
Mu et al. [41] * 24 25 23 26
Park et al. [22] 14 16 15 16
Valentinuzzi et al. [23] 26 27 27 22
Li et al. [27] 20 20 20 20
Mu et al. [24] 27 25 25 26
Mu et al. [21] 27 27 26 26
Zhou et al. [49] 20 24 20 23
Tankyevych et al. [33] 24 25 23 25
Tong et al. [31] 33 21 30 24
Cui et al. [34] 21 21 21 21
Wang et al. [35] 23 18 18 16
Zhao et al. [28] 27 22 22 20
* deep learning analysis.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Radiomics-Clinical AI Model With Probability Weighted Strategy for Prognosis Prediction in Non-small Cell Lung Cancer

Fuk-Hay Tang

et al.

,

2023

Machine Learning Classifiers for Predictive Biomarkers Combining Clinical and Radiomic Data in Testicular Cancer

Catharina Silvia Lisson

et al.

,

2023

Title: Lung-molGPA in EGFR-mutated Adenocarcinoma: Prognostic Implications of Molecular Subtypes and Targeted Therapies

Feng-Che Kuan

et al.

,

2020

Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated