Prerpints.org logo
Preprint
Communication

Ulcerative Colitis, LAIR1 and TOX2 expression and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks

Altmetrics

Downloads

118

Views

67

Comments

0

A peer-reviewed article of this preprint also exists.

supplementary.txt (222.19KB )

Submitted:

02 November 2024

Posted:

05 November 2024

You are already at the latest version

Alerts
Abstract
Background: Ulcerative colitis is a chronic inflammatory bowel disease of the colon mucosa associated with a higher risk of colorectal cancer. Objective: This study is a proof-of-concept analysis aimed to test the feasibility of classifying hematoxylin and eosin (H&E) histological images of ulcerative colitis, normal colon, and colorectal cancer using Artificial Intelligence (AI); Methods: A convolutional neural network (CNN) was designed and trained to classify the 3 types of diagnosis, including 35 cases of ulcerative colitis (n = 9,281 images), 21 colon control (n = 12,246), and 18 colorectal cancer (n = 63,725). The data were partitioned into training (70%) and validation sets (10%) for training the network, and a test set (20%) to test the performance on the new data. Explainable artificial intelligence for computer vision was performed using gradient-weighted class activation mapping (Grad-CAM) to understand the classification decisions; and additional LAIR1 and TOX2 immunohistochemistry was performed in ulcerative colitis to analyze the immune microenvironment; Results: The trained network classified the 3 diagnoses with high performance. The model classified ulcerative colitis with an accuracy of 99.1%, precision of 97.1%, recall of 94.8%, an F1-score of 95.9%, and specificity of 99.6%. For colorectal cancer, the performance was as follows: accuracy, 99.8%; precision, 99.9%; recall, 99.8%; F1-score 99.9%; and specificity, 99.7%. The Grad-CAM heatmap confirmed which parts of the image were most important for classification. The CNN also managed to differentiate between steroid-requiring (SR) and nonsteroid requiring (non-SR) ulcerative colitis based on H&E, LAIR1 and TOX2 staining; Conclusions: Convolutional neural networks are essential tools for deep learning and are especially suited for image recognition in conditions such as ulcerative colitis and colorectal cancer; LAIR1 and TOX2 are promising immuno-oncology markers in ulcerative colitis.
Keywords: 
Subject: Medicine and Pharmacology  -   Pathology and Pathobiology

1. Introduction

Inflammatory bowel disease is a chronic inflammatory condition characterized by relapsing-remitting inflammation of the gastrointestinal tract. It includes two main entities: ulcerative colitis, which affects the colon, and Crohn’s disease, which can involve any part of the gastrointestinal tract [1]. The estimated prevalence of inflammatory bowel disease is up to 0.8% in countries such as the UK [2]. The pathogenesis of this condition is still not well understood in both entities, and there is overlap between them [3]. In pathogenesis, both host immune response and microbial factors are involved, including alterations of the epithelial barrier [4], dysregulation of immune cells, dysregulation of secreted mediators, microbes, and genetic susceptibility [5] (Table 1).
Ulcerative colitis is an inflammatory disease limited to the colon that usually affects the rectum and extends to the proximal side. The prevalence of ulcerative colitis is estimated to be 5 million cases worldwide [45]. The onset of the disease is usually gradual and progressive over time, and is typically accompanied by diarrhea. Systemic symptoms include weight loss, fatigue, and fever [5,45]. The disease severity ranges from mild, to moderate, and severe, and the Mayo scoring system/Disease Activity Index (DAI) can be used to assess disease severity and monitor response to therapy [46,47]. The variables of the Mayo score are stool pattern, most severe rectal bleeding of the day, endoscopic findings, and global assessment by clinician [48].
Acute complications of ulcerative colitis include severe bleeding that occurs in up to 10% of patients, fulminant colitis and toxic megacolon, and perforation [5,45]. Ulcerative colitis is primary a disease of the colon. However, extraintestinal manifestations are also present: musculoskeletal (arthritis and arthropathy), eye (uveitis and episcleritis), skin (erythema nodosum and pyoderma gangrenosum), hepatobiliary (primary sclerosing cholangitis, fatty liver and autoimmune liver disease), hematopoietic/coagulation (thromboembolism), and pulmonary [5,45,49,50,51,52,53,54].
The diagnosis of ulcerative colitis is based on the presence of chronic diarrhea for > 4 weeks and evidence of colon inflammation on histological biopsy [45]. Since these characteristics are not specific, other diseases must be excluded before reaching the diagnosis of ulcerative colitis, including Crohn’s disease, infection colitis, radiation colitis, diverticulitis, diversion colitis, solitary rectal ulcer syndrome, graft-versus-host disease, and medication-associated colitis [5]. Of note, patients with ulcerative colitis have a higher risk of developing dysplasia and colorectal cancer [5]. The extent of the affected area (patients with pancolitis) and duration of the disease are the two major risk factors associated with the development of neoplasia [55].
The aim of treatment in patients with active ulcerative colitis is to achieve clinical and endoscopic remission. Mesalamine (mesalazine) is a 5-aminosalicylic acid derivative is initially used. If there is no improvement, glucocorticoids are used (for example, budesonide, or prednisone). Failure to respond and refractory disease may require systemic glucocorticoids and biological agents, such as anti-tumor necrosis (TNF) agents, anti-α4β7-integrin (vedolizumab), anti-interleukin antibody-based therapy, sphingosine-1-phosphate (S1P) receptor modulators, or small molecules (tofacitinib, an small-molecule Janus kinase inhibitor) [56,57,58,59,60,61,62,63,64,65,66].
Colorectal cancer (CRC) is a common and lethal disease, with an annual incidence of approximately 153,000 cases [67]. The risk of CRC depends on environmental and genetic factors such as inflammatory bowel disease [68]. The diagnosis is usually made by colonoscopy. The management of localized disease is surgical resection and adjuvant chemotherapy [69].
Convolutional neural networks for deep learning image classification is an application of digital pathology and artificial intelligence in translational medicine and clinical practice [70]. The deep learning workflow includes data preprocessing, network building, training, network performance improvement by tuning hyperparameters or running multiple trials, and visualization and verification of network behavior during and after training [71].
This study used several convolutional neural networks to classify images of ulcerative colitis and differentiate between colonic control and colorectal cancer (adenocarcinoma). In addition, the protein expression of a new immuno-oncology maker was explored in the ulcerative colitis cases.

2. Materials and Methods

Hematoxylin and eosin (H&E) histological slides of 35 ulcerative colitis patients were retrieved from our previous publication [72]. All patients were started on 5-ASA (mesalazine) treatment with or without probiotics. A more aggressive treatment (prednisolone as the first choice) was used if the initial treatment failed to induct remission state (UC-DAI score, 1–2) or if the disease relapsed as defined by symptoms (UC-DAI > 2, with bloody stool) as well as by laboratory data and/or colonoscopy [72]. After a follow-up of 2 years, 35 cases were classified into two groups: 22 cases of mesalazine-responsive ulcerative colitis (clinically denominated as “benign”) and 13 cases of steroids-requiring ulcerative colitis (clinically characterized by a more “aggressive” behavior).
Other recorded clinicopathological characteristics included age at biopsy, sex, biopsy location, Baron score for endoscopic grading of ulcerative colitis [73,74,75], and histopathological Geboes score for ulcerative colitis [76] (Table 2). This project was in accordance with the World Medical Association (WMA) Declaration of Helsinki on ethical principles for medical research involving human subjects (IRB 13R-119).
Immunohistochemistry targeting LAIR1 was performed using a Bond-Max fully automated immunohistochemistry and in-situ hybridization staining system following the manufacturer’s instructions (Leica Biosystems K.K., Tokyo, Japan). For immunohistochemistry using DAB chromogen and hematoxylin counterstain, polymer detection system DS9800 was used (Leica Byosystems). The primary antibody targeted leukocyte-associated immunoglobulin-like receptor (LAIR1/CD305) created by Dr. Giovanna Roncador of Monoclonal Antibodies Core Unit, located a Spanish National Cancer Research Center (also known as CNIO: Centro Nacional de Investigaciones Oncológicas). The LAIR1 is a rat monoclonal antibody, clone JAVI82A, antigen used: RBL-1-LAIR1-MYC-DDK transfected cells and last booster with LAIR1 recombinant (Gln22-His163, with a C-terminal 6-His tag); isotype IgG2a; reactivity, human; localization, membrane.
TOX2 antibody targeted TOX High Mobility Group Box Family Member 2, and was also developed by CNIO. Properties: clone name TOM924D, rat monoclonal, IgG2b K, antigen HIS-SUMO-hTOX2-Strep-tag2 full length protein, human reactivity, nucleus localization.
The slides were scanned using a NanoZoomer S360 virtual slide scanner (#C13220-01, Hamamatsu Photonics, Hamamatsu, Japan). Digital images were visualized using NDP.view2 software (#U12388-01, Hamamatsu Photonics), and each intestinal biopsy was exported into a jpeg file at 200× magnification and 150 dpi.
Images were cropped at 243×243 size, reviewed by the pathologist (J.C.), and unproductive images were excluded from the analysis. The filtering criteria were as follows: (1) cropped images of not 243×243 size; (2) images less than 5-31 KB that usually do not contain tissue; (3) images that did not contain diagnostic areas based on histopathological criteria; (4) images with artifacts such as broken and folded tissue and wrongly stained. The series also included 18 cases of colorectal cancer (adenocarcinoma) and 21 patients of colon control who underwent diagnostic biopsies and surgical resection specimens.
A convolutional neural network (CNN) was designed based on transfer learning from ResNet-18; and trained to classify the 3 types of images, ulcerative colitis (n = 9,281), colon control (n = 12,246), and colorectal cancer (n = 63,725). The data were partitioned into a training set (70% of the images) to train the network, a validation set (10%) to test the performance of the network during training, and a test set (20%) as a holdout (new data) to test the performance on new data. The order of the images was randomized to ensure that the learning of classes evenly.
Data normalization was applied to the input images as previously described [71]. The code was run in Matlab programing language and numeric computing environment (R2023b, update 9 released 30 Jul 2024, MathWorks, Tokyo, Japan), as we have recently described [71]. The input size was 224-by-224 (224×224×3). In summary, the code loaded the pretrained CNN, replaced the final layers, trained the network, made predictions, assessed the accuracy, and deployed the results. The design and training parameters of the CNN are listed in Table 3.
Advanced explainable artificial intelligence for computer vision was performed to understand the classification decisions by the deep learning network using the gradient-weighted class activation mapping (Grad-CAM) heatmap technique.
The performance of the ResNet-18-based network was compared to other pre-trained networks, including AlexNet, DenseNet-201, EfficientNet-b0, GoogLeNet, Inception-v3, MobileNet-v2, NASNet-Large, NASNet-Mobile, ResNet-18, ResNet-50, ResNet-101, ShuffleNet, VGG-16, VGG-19, and Xception (Table 4).
Preprints 138358 g001
Figure 1. The 3 types of samples included in this study: ulcerative colitis (left), colorectal cancer (adenocarcinoma, middle), and colon control (right). Original magnification 100×.
Figure 1. The 3 types of samples included in this study: ulcerative colitis (left), colorectal cancer (adenocarcinoma, middle), and colon control (right). Original magnification 100×.
Preprints 138358 g001
Preprints 138358 g002
Figure 2. Cropped images of ulcerative colitis. Original magnification 200×. Images were cropped at 224 × 224 size. Ulcerative colitis is an idiopathic chronic inflammation that affects the colon mucosa. This disorder characteristically affects the rectum and extends toward proximal sections of the colon in a continuous manner. Microscopically, there are signs of active chronic colitis when untreated. Chronicity includes distorted architecture of the crypts, such as atrophy, irregular spacing, shortening, and branching; inflammation of the lamina propria with basal lymphoplasmacytosis, and Panet cell metaplasia or hyperplasia. Disease activity is confirmed by neutrophil infiltration of the muscosa, cryptitis, crypt abscess, or ulceration. Typically, inflammation is limited to the mucosa and submucosa, and granulomas and fissuring ulcers are absent. Well-established disease can be associated with dysplasia of the epithelium, either low-grade, or high-grade. The most commonly used score to evaluate the histological features is the Geboes Score [89,90,91,92,93].
Figure 2. Cropped images of ulcerative colitis. Original magnification 200×. Images were cropped at 224 × 224 size. Ulcerative colitis is an idiopathic chronic inflammation that affects the colon mucosa. This disorder characteristically affects the rectum and extends toward proximal sections of the colon in a continuous manner. Microscopically, there are signs of active chronic colitis when untreated. Chronicity includes distorted architecture of the crypts, such as atrophy, irregular spacing, shortening, and branching; inflammation of the lamina propria with basal lymphoplasmacytosis, and Panet cell metaplasia or hyperplasia. Disease activity is confirmed by neutrophil infiltration of the muscosa, cryptitis, crypt abscess, or ulceration. Typically, inflammation is limited to the mucosa and submucosa, and granulomas and fissuring ulcers are absent. Well-established disease can be associated with dysplasia of the epithelium, either low-grade, or high-grade. The most commonly used score to evaluate the histological features is the Geboes Score [89,90,91,92,93].
Preprints 138358 g002
Preprints 138358 g003
Figure 3. Cropped images of colorectal cancer (adenocarcinoma). Original magnification 200×. Images were cropped at 224 × 224 size. Adenocarcinoma of the colon is a glandular neoplasm that accounts for approximately 98% of all colonic cancers. Patients with inflammatory bowel disease, polyposis, and Lynch syndrome [94,95] are at higher risk of developing colorectal cancer. Most cases display well- or moderate differentiation of carcinoma glands accompanied by marked growth of fibrous connective tissue known as desmoplasia [96,97]. The glands can show a cribriform pattern and are filled with necrotic debris. Adenocarcinomas are characterized by epithelial cells with stretched and stratified nuclei, which create complex glandular structures. The nuclei exhibit polymorphism and loss of polarity. The tumor immune microenvironment exhibits variable infiltration of inflammatory cells. There are several recognized subtypes, including adenoma-like, adenosquamous, mucinous, micropapillary, signet-ring, serrated, and sarcomatoid [89,97,98,99,100,101,102,103].
Figure 3. Cropped images of colorectal cancer (adenocarcinoma). Original magnification 200×. Images were cropped at 224 × 224 size. Adenocarcinoma of the colon is a glandular neoplasm that accounts for approximately 98% of all colonic cancers. Patients with inflammatory bowel disease, polyposis, and Lynch syndrome [94,95] are at higher risk of developing colorectal cancer. Most cases display well- or moderate differentiation of carcinoma glands accompanied by marked growth of fibrous connective tissue known as desmoplasia [96,97]. The glands can show a cribriform pattern and are filled with necrotic debris. Adenocarcinomas are characterized by epithelial cells with stretched and stratified nuclei, which create complex glandular structures. The nuclei exhibit polymorphism and loss of polarity. The tumor immune microenvironment exhibits variable infiltration of inflammatory cells. There are several recognized subtypes, including adenoma-like, adenosquamous, mucinous, micropapillary, signet-ring, serrated, and sarcomatoid [89,97,98,99,100,101,102,103].
Preprints 138358 g003
Preprints 138358 g004
Figure 4. Cropped images of colon control. Original magnification 200×. Images were cropped at 224 × 224 size. The colonic mucosa has the function of absorbing water and electrolytes, which is performed by absorptive columnar cells, and producing of mucus to lubricate, which is produced by globet cells. The mucosa is composed of the epithelium, lamina propria, and muscularis mucosa. The epithelium invaginates and forms glands (crypts), where at its base there is also the presence of enteroendocrine, Paneth cells, and stem cells. The lamina propria is rich in capillaries and lymphatics. Loose connective tissue and nerve plexuses are found in the submucosa. The muscularis propria has an inner circular layer and an outer longitudinal layer, and within them, the Auerbach nerve plexus is located. The outer layers are the subserosa and serosa [89,104,105].
Figure 4. Cropped images of colon control. Original magnification 200×. Images were cropped at 224 × 224 size. The colonic mucosa has the function of absorbing water and electrolytes, which is performed by absorptive columnar cells, and producing of mucus to lubricate, which is produced by globet cells. The mucosa is composed of the epithelium, lamina propria, and muscularis mucosa. The epithelium invaginates and forms glands (crypts), where at its base there is also the presence of enteroendocrine, Paneth cells, and stem cells. The lamina propria is rich in capillaries and lymphatics. Loose connective tissue and nerve plexuses are found in the submucosa. The muscularis propria has an inner circular layer and an outer longitudinal layer, and within them, the Auerbach nerve plexus is located. The outer layers are the subserosa and serosa [89,104,105].
Preprints 138358 g004

3. Results

3.1. Image classification based on transfer learning from ResNet-18

Transfer learning using the pretrained ResNet-18 CNN was used to classify images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control. The data were partitioned into a training set (70% of the images) to train the network, a validation set (10%) to test the performance of the network during training, and a test set (20%) as a holdout (new data) to test the performance on new data. The network performance during training and validation is shown in Figure 5. The network achieved high accuracy and low loss during the first 100 iterations.
After training, new images of the test set (holdout) were classified using the trained CNN. The result achieved a performance of 99%. The confusion matrix is shown in Figure 6.
The classification performance for each diagnosis is presented in Table 5.

3.2. Grad-CAM Heatmap Analysis

Grad-CAM heatmap was used to visualize which parts of the images were important to the classification decision of the network. This method uses the classification score gradient relative to the final convolutional feature map. The parts of images with large values for the Grad-CAM maps had the greatest effect on the network score for that diagnosis (image classification). Some examples of correctly classified images are shown in Figure 7; the network was focusing on the epithelial layer and inflammatory components of the lamina propria.
In a few cases, the classification of images was incorrect and there was a discrepancy between diagnosis and prediction. A review of these cases showed that in most cases, the reason for discrepancy was that the image was not diagnostic from a histopathological point of view or that the network was focusing on an incorrect part of the image to make the classification, as shown in the Grad-CAM analysis (Figure 8).

3.3. Differentiation between steroid-requiring (SR) and nonsteroid requiring (non-SR) ulcerative colitis

Ulcerative colitis can be divided into two clinical groups based on the requirement for steroids to control colon inflammation. Transfer learning from ResNet-18 was performed to predict and classify the H&E images of ulcerative colitis into the two subtypes of steroid-requiring (“aggressive”) and mesalazine-responsive (“benign”). In the test set, the accuracy was 79.53%. The confusion matrix is shown in Figure 9, and the network performand in Table 6.

3.4. Differentiation Between Steroid-Requiring (SR) and Nonsteroid Requiring (non-SR) Ulcerative Colitis Using LAIR1 Immunohistochemistry

Ulcerative colitis can be divided into two clinical groups based on the requirement for steroids to control colon inflammation as described in section 3.3. The analysis of H&E highlighted the importance of both epithelial and inflammatory components. LAIR1 is a new marker of the immune microenvironment and an immuno-oncology target. Therefore, transfer learning from ResNet-18 was performed to predict and classify the LAIR1 images of ulcerative colitis into the two subtypes of steroid-requiring (“aggressive”) and mesalazine-responsive (“benign”). In the test set, the accuracy was 88.31%. The confusion matrix is shown in Figure 10, and network performance in Table 7.
Examples of steroid-requiring and mesalazine-response ulcerative colitis and characteristic LAIR1 immunohistochemistry is shown in Figure 11. Evaluation of the whole-tissue images of LAIR1 pointed out that steroid-requiring had a more prominent inflammation of the lamina propria as shown in Figure 11 and Figure 12.

3.5. Differentiation Between Steroid-Requiring (SR) and Nonsteroid Requiring (non-SR) Ulcerative Colitis Using TOX2 Immunohistochemistry

TOX2 (TOX high mobility group box family member 2) is a new marker of the immune microenvironment and an immuno-oncology target. Transfer learning using ResNet-18 classified TOX2 images of ulcerative colitis into steroid-requiring and mesalazine-responsive. In the test set, the accuracy was 85.62%. The confusion matrix is shown in Figure 13, and network performance in Table 8.
Examples of steroid-requiring and mesalazine-response ulcerative colitis and characteristic TOX2 immunohistochemistry is shown in Figure 14. Evaluation of the whole-tissue images of TOX2 pointed out that TOX2-positive inflammatory component was higher in mesalazine-responsive cases as shown in Figure 14 and 15.

3.6. Image Classification Using Transfer Learning with Other Convolutional Neural Networks

The performance of ResNet-18 using H&E images was compared with other CNNs. Transfer learning using several pretrained CNN was used to classify images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control. The network performance of the validation test set (holdout, new data) is shown in Table 9 and Appendix A. ResNet-18 provided the best performance with an accuracy of 99%. However, when all CNNs were run, the best results were obtained using DenseNet-201 (99.3%), ResNet-50 (99.14%), Inception-v3 (99.13%), and ResNet-101 (99.1%); however, these CNNs required longer training times. Of note, the NasNet-Large CNN required a longer training time (12495 min 37 sec).

4. Discussion

Ulcerative colitis is a chronic inflammatory bowel disease that primary affects the mucosal layer of the colon recurrently. It usually affects the rectum and extends continuously toward the proximal segments of the colon [5,106]. Patients usually present with diarrhea that may include blood, with a gradual and progressive onset of symptoms [5,45,106]. Disease evaluation includes history, laboratory studies, endoscopy, and biopsy. Several factors affect the disease course, including age at diagnosis, mucosal healing, extension of colitis, and smoking. Chronic complications include stricture, dysplasia, and colorectal cancer [49].
The endoscopic findings are nonspecific and include loss of vascular marking, granular mucosa, petechiae, exudates, edema, erosions, friability, ulcerations, and bleeding [107]. The endoscopic biopsies show neutrophilic infiltration with cryptitis, crypt abscesses, and ulcerations when the disease is active. Chronic disease involves crypt architectural alterations, chronic inflammation of the lamina propria with lymphoplasmocytosis, and Paneth cell metaplasia or hyperplasia [89,90,104]. This study analyzed images of ulcerative colitis using convolutional neural network (CNN).
A CNN is a deep learning machine learning technique trained using a large number of images. CNN was used to analyze whole slide images (WSI) from hematoxylin & eosin (H&E) microscope pathology slides of ulcerative colitis, colon control, and colorectal cancer (adenocarcinoma). Artificial intelligence (AI) is the process of being capable of simulating human intelligence, to mimic the cognitive functions of learning and problem solving. AI can be divided into artificial general intelligence, or strong AI, that has generalized human cognitive abilities [108,109,110]; and narrow AI, or weak AI, that focuses on a specific task of human intelligence [111,112]. There are two types of narrow AI: rule-based AI is provided with machine rules to follow, and example-based AI is provided with examples to learn from [113]. This study was a histological analysis using narrow AI to classify the gut images. The slides were digitalized using a slide scanner that converted glass slides into digital data. In this study, both endoscopic biopsy and surgical rejection specimens were used. The CNN could differentiate the 3 types of diagnoses with high performance (99%). However, the differential diagnosis of ulcerative colitis must be considered.
The evaluation and establishment of a diagnosis of ulcerative colitis need to exclude other causes of colitis using history, laboratory studies, endoscopic images, and colon biopsies. In the history, other causes of colitis should be excluded, including parasitic infections, sexually transmitted infections (Neisseria gonorrhoeae and herpes simplex virus), atherosclerotic disease (chronic colonic ischemia), abdominal/pelvic radiation, and use of non-steroidal anti-inflammatory drugs. The stool should also be tested for C. trachomatis, N. gonorrhoeae, HSV, and Treponema pallidum. The endoscopic and histological findings of ulcerative colitis are not specific. However, differentiating ulcerative colitis from Crohn’s disease is important because of its different prognoses and treatments [114]. A comprehensive review and update on ulcerative colitis, including an extended differential diagnosis, was described by Gajendran M. et al. [115]. Among the relevant diseases, neoplasm, Crohn’s disease, and Celiac disease were highlighted [115]. In this study, the image classification comprised cases of ulcerative colitis and colorectal cancer (adenocarcinoma). We recently performed histological image classification of Celiac disease, small intestine control, unspecific duodenal inflammation, and Crohn’s disease [71]. Narrow artificial intelligence (AI) is designed to perform tasks that typically require human intelligence; however, it operates within limited constraints and is task-specific [71]. In future, integrated analysis can performed.
Computer vision provides a series of algorithms for visual inspection, object detection, and tracking, and feature detection, extraction, and matching. Pretrained object detection can be performed using the YOLO, SSD, and ACF algorithms. Semantic and instance segmentation using U-Net, SOLO, and the Mask R-CNN [116]. Image classification can be performed using vision transformers, such as ViT. In this study, we employed pretrained image classification neural networks. These pretrained networks have already learned to extract the characteristics of natural images. Therefore, we used them as a starting point to learn the new task of classifying ulcerative colitis, colonic control, and colorectal cancer (adenocarcinoma). Several CNNs were used, and the most accurate ones were DenseNet-201 (99.30% accuracy), ResNet-50 (99.14%), Inception-v3 (99.13%), ResNet-101 (99.1%), and ResNet-18 (99%). The most relevant features of a CNN are accuracy, speed, and size. In this study, ResNet-18 was the most convenient CNN to use due to its accuracy and relative prediction time using GPU. NASNet-Large provided good performance; however, despite having high accuracy, the prediction time was the slowest.
Explainable AI (XAI) allows humans to understand and trust results by providing clear and understandable explanations [117,118,119,120]. In computer vision, a common method to demonstrate XAI techniques is to overlay an explanation on the image using an explanation heatmap. In this study, grad-CAM was used [121,122]. The other XAI methods are LIME and SHAP [123]. Grad-CAM uses a gradient-weighted class activation mapping technique to understand why a deep learning network makes classification decisions. As shown in the results section, the Grad-CAM analysis confirmed that the CNN focused on the correct area of the images to perform classification. Interestingly, in the discordant cases, the Grad-CAM analysis revealed incorrect focus by the CNN. In addition, some discordant cases were due to nondiagnostic images.
In the last 5 years there have been several reports on ulcerative colitis and deep learning. Most applications have focused on endoscopic image analysis [124,125,126,127,128,129,130,131]. However, some research has been published regarding histological images. Dawid Rymarczk et al. used deep learning models to analyze the histological disease activity in Crohn’s disease and ulcerative colitis [132]. Niels Vande Casteele et al. used a deep learning algorithm to identify eosinophils in colonic biopsies of active ulcerative colitis [133]. Jun Ohara et al. applied deep learning to detect goblet cell mucus [134]. Laurent Peyrin-Biroulet used an AI algorithm to measure the Nancy index in histological images of patients with ulcerative colitis [135]. Our study is different from previous publications because made diagnosis of ulcerative colitis against colon control and identified images of colorectal cancer. To the best of our knowledge, this type of study has not been conducted before now.
LAIR1 protein functions an inhibitory receptor that is expressed in monocytes, natural killer cells, T and B-lymphocytes. LAIR1 is also expressed in macrophages and regulates their activation [136]. In lymphoma, LAIR1 expression by tumor-associated macrophages has been described [137]. In ulcerative colitis, the expression of LAIR1 has been recently described by Mina Hassan-Zahraee et al. [138]. However, to the best of our knowedge, LAIR1 has not been described in other inflammatory bowel diseases. In our study, LAIR1 was expressed by cells of the immune microenvironment, and the CNN managed to classify between steroid-requiring (SR) and nonsteroid requiring (non-SR) ulcerative colitis. Therefore, LAIR1 is a promising immuno-oncology marker in ulcerative colitis.
TOX2 (TOX High Mobility Group Box Family Member 2) is a transcription factor related to T cell exhaustion [139], which is a broad term used to describe the T cell functions in situation of chronic antigen stimulation, such as inflammatory bowel disease and response to tumors [140]. Evidence indicates that TOX2 is expressed in T follicular helper (TFH) cells, similar to PD-1 marker, and that would suppress CD4 cytotoxic T cell differentiation [141]. TOX2 has been related to the survival of patients with acute myeloid leukemia [140], and to the pathogenesis of atopic dermatitis [142]. Because of the relationship with PD-1, TOX2 is a promising immuno-oncology marker in ulcerative colitis as well.

5. Conclusions

A convolutional neural network predicts ulcerative colitis, colon control, and colorectal cancer (adenocarcinoma) with good performance.

Supplementary Materials

The following supporting information can be downloaded at the website of this paper posted on Preprints.org. Training and testing of neural networks (Text file 1 and 2).

Author Contributions

Conceptualization, J.C.; methodology, J.C., G.R., R.H.; software, J.C.; formal analysis, J.C.; primary antibodies, J.C. and G.R.; writing—original draft preparation, J.C.; writing—review, J.C., and R.H.; funding acquisition, J.C. and R.H. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Ministry of Education, Culture, Sports, Science and Technology of Japan, KAKEN grants 23K06454, 18K15100, and 15K19061. R.H. is funded by University of Sharjah (grant no: 24010902153) and ASPIRE, the technology program management pillar of Abu Dhabi’s Advanced Technology Research Council (ATRC), via the ASPIRE Precision Medicine Research Institute Abu Dhabi (VRI-20–10).

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of TOKAI UNIVERSITY, SCHOOL OF MEDICINE (protocol code IRB14R-080, IRB20-156, and 13R-119).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

All the data, including methodology, are available upon request to Dr. Joaquim Carreras (joaquim.carreras@tokai.ac.jp).

Conflicts of Interest

The authors declare no conflicts of interest.

Appendix A. Confusion matrices (test set, new data) (Accuracy)

DenseNet-201 (99.3%)
Adenocarcinoma 12741 3 0
Colon control 1 2362 29
Ulcerative colitis 3 84 1827
Adenocarcinoma Colon control Ulcerative colitis
ResNet-50 (99.14%)
Adenocarcinoma 12740 12 4
Colon control 3 2355 44
Ulcerative colitis 2 82 1808
Adenocarcinoma Colon control Ulcerative colitis
Inception-v3 (99.13%)
Adenocarcinoma 12737 12 5
Colon control 3 2350 37
Ulcerative colitis 5 87 1814
Adenocarcinoma Colon control Ulcerative colitis
ResNet-101 (99.1%)
Adenocarcinoma 12740 11 8
Colon control 0 2359 51
Ulcerative colitis 5 79 1797
Adenocarcinoma Colon control Ulcerative colitis
ResNet-18 (99%)
Adenocarcinoma 12732 13 2
Colon control 5 2345 52
Ulcerative colitis 8 91 1802
Adenocarcinoma Colon control Ulcerative colitis
ShuffleNet (98.94%)
Adenocarcinoma 12734 14 8
Colon control 7 2336 49
Ulcerative colitis 4 99 1799
Adenocarcinoma Colon control Ulcerative colitis
MobileNet-v2 (98.89%)
Adenocarcinoma 12734 19 24
Colon control 7 2334 40
Ulcerative colitis 4 96 1792
Adenocarcinoma Colon control Ulcerative colitis
NasNet-Large (98.88%)
Adenocarcinoma 12732 23 8
Colon control 5 2287 8
Ulcerative colitis 8 139 1840
Adenocarcinoma Colon control Ulcerative colitis
GoogLeNet-Places365 (98.86%)
Adenocarcinoma 12702 4 0
Colon control 24 2332 35
Ulcerative colitis 19 113 1821
Adenocarcinoma Colon control Ulcerative colitis
VGG-19 (98.8%)
Adenocarcinoma 1270 19 13
Colon control 3 2371 109
Ulcerative colitis 2 59 1734
Adenocarcinoma Colon control Ulcerative colitis
EfficientNet-b0 (98.79%)
Adenocarcinoma 12733 8 8
Colon control 4 2311 49
Ulcerative colitis 8 130 1799
Adenocarcinoma Colon control Ulcerative colitis
AlexNet (98.77%)
Adenocarcinoma 12740 18 12
Colon control 2 2302 46
Ulcerative colitis 3 129 1798
Adenocarcinoma Colon control Ulcerative colitis
Xception (98.66%)
Adenocarcinoma 12718 30 12
Colon control 16 2302 43
Ulcerative colitis 11 117 1801
Adenocarcinoma Colon control Ulcerative colitis
VGG-16 (98.65%)
Adenocarcinoma 12744 26 59
Colon control 1 2363 85
Ulcerative colitis 0 60 1712
Adenocarcinoma Colon control Ulcerative colitis
GoogLeNet (98.6%)
Adenocarcinoma 12726 23 9
Colon control 5 2256 17
Ulcerative colitis 14 170 1830
Adenocarcinoma Colon control Ulcerative colitis
NasNet-Mobile (98.58%)
Adenocarcinoma 12724 18 4
Colon control 17 2320 88
Ulcerative colitis 4 111 1764
Adenocarcinoma Colon control Ulcerative colitis

References

  1. Guan, Q. A Comprehensive Review and Update on the Pathogenesis of Inflammatory Bowel Disease. J. Immunol. Res. 2019, 2019, 7247238. [Google Scholar] [CrossRef] [PubMed]
  2. Jones, G.R.; Lyons, M.; Plevris, N.; Jenkinson, P.W.; Bisset, C.; Burgess, C.; Din, S.; Fulforth, J.; Henderson, P.; Ho, G.T.; et al. IBD prevalence in Lothian, Scotland, derived by capture-recapture methodology. Gut 2019, 68, 1953–1960. [Google Scholar] [CrossRef] [PubMed]
  3. Porter, R.J.; Arends, M.J.; Churchhouse, A.M.D.; Din, S. Inflammatory Bowel Disease-Associated Colorectal Cancer: Translational Risks from Mechanisms to Medicines. J. Crohns Colitis 2021, 15, 2131–2141. [Google Scholar] [CrossRef] [PubMed]
  4. Michielan, A.; D’Inca, R. Intestinal Permeability in Inflammatory Bowel Disease: Pathogenesis, Clinical Evaluation, and Therapy of Leaky Gut. Mediators Inflamm. 2015, 2015, 628157. [Google Scholar] [CrossRef] [PubMed]
  5. Scott B Snapper, Clara Abraham. Immune and microbial mechanisms in the pathogenesis of inflammatory bowel disease. In: UpToDate, Sunanda V Kane and Kristen M Robson (Ed), Wolters Kluwer. Accessed September 21, 2024.
  6. Kudelka, M.R.; Stowell, S.R.; Cummings, R.D.; Neish, A.S. Intestinal epithelial glycosylation in homeostasis and gut microbiota interactions in IBD. Nat. Rev. Gastroenterol. Hepatol. 2020, 17, 597–617. [Google Scholar] [CrossRef]
  7. Png, C.W.; Linden, S.K.; Gilshenan, K.S.; Zoetendal, E.G.; McSweeney, C.S.; Sly, L.I.; McGuckin, M.A.; Florin, T.H. Mucolytic bacteria with increased prevalence in IBD mucosa augment in vitro utilization of mucin by other bacteria. Am. J. Gastroenterol. 2010, 105, 2420–2428. [Google Scholar] [CrossRef]
  8. Yao, Y.; Kim, G.; Shafer, S.; Chen, Z.; Kubo, S.; Ji, Y.; Luo, J.; Yang, W.; Perner, S.P.; Kanellopoulou, C.; et al. Mucus sialylation determines intestinal host-commensal homeostasis. Cell 2022, 185, 1172–1188.e1128. [Google Scholar] [CrossRef]
  9. Amoroso, C.; Perillo, F.; Strati, F.; Fantini, M.C.; Caprioli, F.; Facciotti, F. The Role of Gut Microbiota Biomodulators on Mucosal Immunity and Intestinal Inflammation. Cells 2020, 9. [Google Scholar] [CrossRef]
  10. de Souza, H.S.; Fiocchi, C. Immunopathogenesis of IBD: current state of the art. Nat. Rev. Gastroenterol. Hepatol. 2016, 13, 13–27. [Google Scholar] [CrossRef]
  11. Kong, L.; Pokatayev, V.; Lefkovith, A.; Carter, G.T.; Creasey, E.A.; Krishna, C.; Subramanian, S.; Kochar, B.; Ashenberg, O.; Lau, H.; et al. The landscape of immune dysregulation in Crohn’s disease revealed through single-cell transcriptomic profiling in the ileum and colon. Immunity 2023, 56, 444–458. [Google Scholar] [CrossRef]
  12. Lee, S.H.; Kwon, J.E.; Cho, M.L. Immunological pathogenesis of inflammatory bowel disease. Intest. Res. 2018, 16, 26–42. [Google Scholar] [CrossRef] [PubMed]
  13. Mitsialis, V.; Wall, S.; Liu, P.; Ordovas-Montanes, J.; Parmet, T.; Vukovic, M.; Spencer, D.; Field, M.; McCourt, C.; Toothaker, J.; et al. Single-Cell Analyses of Colon and Blood Reveal Distinct Immune Cell Signatures of Ulcerative Colitis and Crohn’s Disease. Gastroenterology 2020, 159, 591–608.e510. [Google Scholar] [CrossRef] [PubMed]
  14. Ramos, G.P.; Papadakis, K.A. Mechanisms of Disease: Inflammatory Bowel Diseases. Mayo Clin. Proc. 2019, 94, 155–165. [Google Scholar] [CrossRef] [PubMed]
  15. Rosen, M.J.; Dhawan, A.; Saeed, S.A. Inflammatory Bowel Disease in Children and Adolescents. JAMA Pediatr. 2015, 169, 1053–1060. [Google Scholar] [CrossRef]
  16. Xavier, R.J.; Podolsky, D.K. Unravelling the pathogenesis of inflammatory bowel disease. Nature 2007, 448, 427–434. [Google Scholar] [CrossRef] [PubMed]
  17. Xu, X.R.; Liu, C.Q.; Feng, B.S.; Liu, Z.J. Dysregulation of mucosal immune response in pathogenesis of inflammatory bowel disease. World J. Gastroenterol. 2014, 20, 3255–3264. [Google Scholar] [CrossRef]
  18. Del Sordo, R.; Lougaris, V.; Bassotti, G.; Armuzzi, A.; Villanacci, V. Therapeutic agents affecting the immune system and drug-induced inflammatory bowel disease (IBD): A review on etiological and pathogenetic aspects. Clin. Immunol. 2022, 234, 108916. [Google Scholar] [CrossRef]
  19. Dunkin, D.; Mehandru, S.; Colombel, J.F. Immune cell therapy in IBD. Dig. Dis. 2014, 32 Suppl. 1, 61–66. [Google Scholar] [CrossRef]
  20. Elhag, D.A.; Kumar, M.; Saadaoui, M.; Akobeng, A.K.; Al-Mudahka, F.; Elawad, M.; Al Khodor, S. Inflammatory Bowel Disease Treatments and Predictive Biomarkers of Therapeutic Response. Int. J. Mol. Sci. 2022, 23. [Google Scholar] [CrossRef]
  21. Fantini, M.C.; Monteleone, G. Update on the Therapeutic Efficacy of Tregs in IBD: Thumbs up or Thumbs down? Inflamm. Bowel Dis. 2017, 23, 1682–1688. [Google Scholar] [CrossRef]
  22. Griffin, H.; Ceron-Gutierrez, L.; Gharahdaghi, N.; Ebrahimi, S.; Davies, S.; Loo, P.S.; Szabo, A.; Williams, E.; Mukhopadhyay, A.; McLoughlin, L.; et al. Neutralizing Autoantibodies against Interleukin-10 in Inflammatory Bowel Disease. N. Engl. J. Med. 2024, 391, 434–441. [Google Scholar] [CrossRef] [PubMed]
  23. Jiang, P.; Zheng, C.; Xiang, Y.; Malik, S.; Su, D.; Xu, G.; Zhang, M. The involvement of TH17 cells in the pathogenesis of IBD. Cytokine Growth Factor. Rev. 2023, 69, 28–42. [Google Scholar] [CrossRef] [PubMed]
  24. Qiu, B.; Zhang, T.; Qin, X.; Ma, S.; Wang, Q. The immune factors have complex causal regulation effects on inflammatory bowel disease. Front. Immunol. 2023, 14, 1322673. [Google Scholar] [CrossRef] [PubMed]
  25. Yan, J.B.; Luo, M.M.; Chen, Z.Y.; He, B.H. The Function and Role of the Th17/Treg Cell Balance in Inflammatory Bowel Disease. J. Immunol. Res. 2020, 2020, 8813558. [Google Scholar] [CrossRef] [PubMed]
  26. Larabi, A.; Barnich, N.; Nguyen, H.T.T. New insights into the interplay between autophagy, gut microbiota and inflammatory responses in IBD. Autophagy 2020, 16, 38–51. [Google Scholar] [CrossRef]
  27. Lee, M.; Chang, E.B. Inflammatory Bowel Diseases (IBD) and the Microbiome-Searching the Crime Scene for Clues. Gastroenterology 2021, 160, 524–537. [Google Scholar] [CrossRef]
  28. Nishida, A.; Inoue, R.; Inatomi, O.; Bamba, S.; Naito, Y.; Andoh, A. Gut microbiota in the pathogenesis of inflammatory bowel disease. Clin. J. Gastroenterol. 2018, 11, 1–10. [Google Scholar] [CrossRef]
  29. Schirmer, M.; Garner, A.; Vlamakis, H.; Xavier, R.J. Microbial genes and pathways in inflammatory bowel disease. Nat. Rev. Microbiol. 2019, 17, 497–511. [Google Scholar] [CrossRef]
  30. Tarris, G.; de Rougemont, A.; Charkaoui, M.; Michiels, C.; Martin, L.; Belliot, G. Enteric Viruses and Inflammatory Bowel Disease. Viruses 2021, 13. [Google Scholar] [CrossRef]
  31. Zhang, H.; Zhao, S.; Cao, Z. Impact of Epstein-Barr virus infection in patients with inflammatory bowel disease. Front. Immunol. 2022, 13, 1001055. [Google Scholar] [CrossRef]
  32. Zhang, Q.; Chen, D.; Yang, Y. Incidence and Prevalence of Human Papilloma Virus-associated Cancers in IBD. Inflamm. Bowel Dis. 2020, 26, e101. [Google Scholar] [CrossRef] [PubMed]
  33. Beheshti-Maal, A.; Shahrokh, S.; Ansari, S.; Mirsamadi, E.S.; Yadegar, A.; Mirjalali, H.; Zali, M.R. Gut mycobiome: The probable determinative role of fungi in IBD patients. Mycoses 2021, 64, 468–476. [Google Scholar] [CrossRef] [PubMed]
  34. Sokol, H.; Leducq, V.; Aschard, H.; Pham, H.P.; Jegou, S.; Landman, C.; Cohen, D.; Liguori, G.; Bourrier, A.; Nion-Larmurier, I.; et al. Fungal microbiota dysbiosis in IBD. Gut 2017, 66, 1039–1048. [Google Scholar] [CrossRef] [PubMed]
  35. Yan, Q.; Li, S.; Yan, Q.; Huo, X.; Wang, C.; Wang, X.; Sun, Y.; Zhao, W.; Yu, Z.; Zhang, Y.; et al. A genomic compendium of cultivated human gut fungi characterizes the gut mycobiome and its relevance to common diseases. Cell 2024, 187, 2969–2989.e2924. [Google Scholar] [CrossRef]
  36. Doecke, J.D.; Simms, L.A.; Zhao, Z.Z.; Huang, N.; Hanigan, K.; Krishnaprasad, K.; Roberts, R.L.; Andrews, J.M.; Mahy, G.; Bampton, P.; et al. Genetic susceptibility in IBD: overlap between ulcerative colitis and Crohn’s disease. Inflamm. Bowel Dis. 2013, 19, 240–245. [Google Scholar] [CrossRef]
  37. Ye, B.D.; McGovern, D.P. Genetic variation in IBD: progress, clues to pathogenesis and possible clinical utility. Expert. Rev. Clin. Immunol. 2016, 12, 1091–1107. [Google Scholar] [CrossRef]
  38. Vermeire, S. Review article: genetic susceptibility and application of genetic testing in clinical management of inflammatory bowel disease. Aliment. Pharmacol. Ther. 2006, 24 Suppl. 3, 2–10. [Google Scholar] [CrossRef]
  39. Lees, C.W.; Barrett, J.C.; Parkes, M.; Satsangi, J. New IBD genetics: common pathways with other diseases. Gut 2011, 60, 1739–1753. [Google Scholar] [CrossRef]
  40. Peters, L.A.; Perrigoue, J.; Mortha, A.; Iuga, A.; Song, W.M.; Neiman, E.M.; Llewellyn, S.R.; Di Narzo, A.; Kidd, B.A.; Telesco, S.E.; et al. A functional genomics predictive network model identifies regulators of inflammatory bowel disease. Nat. Genet. 2017, 49, 1437–1449. [Google Scholar] [CrossRef]
  41. Graham, D.B.; Xavier, R.J. From genetics of inflammatory bowel disease towards mechanistic insights. Trends Immunol. 2013, 34, 371–378. [Google Scholar] [CrossRef]
  42. Liu, J.Z.; van Sommeren, S.; Huang, H.; Ng, S.C.; Alberts, R.; Takahashi, A.; Ripke, S.; Lee, J.C.; Jostins, L.; Shah, T.; et al. Association analyses identify 38 susceptibility loci for inflammatory bowel disease and highlight shared genetic risk across populations. Nat. Genet. 2015, 47, 979–986. [Google Scholar] [CrossRef] [PubMed]
  43. de Lange, K.M.; Moutsianas, L.; Lee, J.C.; Lamb, C.A.; Luo, Y.; Kennedy, N.A.; Jostins, L.; Rice, D.L.; Gutierrez-Achury, J.; Ji, S.G.; et al. Genome-wide association study implicates immune activation of multiple integrin genes in inflammatory bowel disease. Nat. Genet. 2017, 49, 256–261. [Google Scholar] [CrossRef]
  44. Liu, Z.; Liu, R.; Gao, H.; Jung, S.; Gao, X.; Sun, R.; Liu, X.; Kim, Y.; Lee, H.S.; Kawai, Y.; et al. Genetic architecture of the inflammatory bowel diseases across East Asian and European ancestries. Nat. Genet. 2023, 55, 796–806. [Google Scholar] [CrossRef]
  45. Le Berre, C.; Honap, S.; Peyrin-Biroulet, L. Ulcerative colitis. Lancet 2023, 402, 571–584. [Google Scholar] [CrossRef] [PubMed]
  46. Schroeder, K.W.; Tremaine, W.J.; Ilstrup, D.M. Coated oral 5-aminosalicylic acid therapy for mildly to moderately active ulcerative colitis. A randomized study. N. Engl. J. Med. 1987, 317, 1625–1629. [Google Scholar] [CrossRef]
  47. Travis, S.P.; Schnell, D.; Krzeski, P.; Abreu, M.T.; Altman, D.G.; Colombel, J.F.; Feagan, B.G.; Hanauer, S.B.; Lichtenstein, G.R.; Marteau, P.R.; et al. Reliability and initial validation of the ulcerative colitis endoscopic index of severity. Gastroenterology 2013, 145, 987–995. [Google Scholar] [CrossRef] [PubMed]
  48. Lewis, J.D.; Chuai, S.; Nessel, L.; Lichtenstein, G.R.; Aberra, F.N.; Ellenberg, J.H. Use of the noninvasive components of the Mayo score to assess clinical response in ulcerative colitis. Inflamm. Bowel Dis. 2008, 14, 1660–1666. [Google Scholar] [CrossRef]
  49. Gros, B.; Kaplan, G.G. Ulcerative Colitis in Adults: A Review. JAMA 2023, 330, 951–965. [Google Scholar] [CrossRef] [PubMed]
  50. Yu, Y.R.; Rodriguez, J.R. Clinical presentation of Crohn’s, ulcerative colitis, and indeterminate colitis: Symptoms, extraintestinal manifestations, and disease phenotypes. Semin. Pediatr. Surg. 2017, 26, 349–355. [Google Scholar] [CrossRef]
  51. Conrad, K.; Roggenbuck, D.; Laass, M.W. Diagnosis and classification of ulcerative colitis. Autoimmun. Rev. 2014, 13, 463–466. [Google Scholar] [CrossRef]
  52. Wang, C.R.; Tsai, H.W. Seronegative spondyloarthropathy-associated inflammatory bowel disease. World J. Gastroenterol. 2023, 29, 450–468. [Google Scholar] [CrossRef] [PubMed]
  53. Rawal, K.K.; Shukla, V.P.; Chikani, S.; Thakkar, M.; Ruparelia, M.; Chudasama, R.K. Prevalence of extraintestinal manifestations in ulcerative colitis and associated risk factors. Indian. J. Gastroenterol. 2021, 40, 477–482. [Google Scholar] [CrossRef] [PubMed]
  54. Fine, S. Extraintestinal Manifestations of Inflammatory Bowel Disease. R I Med J (2013) 2022, 105, 13–19. [Google Scholar] [PubMed]
  55. Lutgens, M.W.; van Oijen, M.G.; van der Heijden, G.J.; Vleggaar, F.P.; Siersema, P.D.; Oldenburg, B. Declining risk of colorectal cancer in inflammatory bowel disease: an updated meta-analysis of population-based cohort studies. Inflamm. Bowel Dis. 2013, 19, 789–799. [Google Scholar] [CrossRef]
  56. Steenholdt, C.; Dige Ovesen, P.; Brynskov, J.; Seidelin, J.B. Tofacitinib for Acute Severe Ulcerative Colitis: A Systematic Review. J. Crohns Colitis 2023, 17, 1354–1363. [Google Scholar] [CrossRef]
  57. Lopez-Sanroman, A.; Esplugues, J.V.; Domenech, E. Pharmacology and safety of tofacitinib in ulcerative colitis. Gastroenterol. Hepatol. 2021, 44, 39–48. [Google Scholar] [CrossRef]
  58. Sandborn, W.J.; Su, C.; Sands, B.E.; D’Haens, G.R.; Vermeire, S.; Schreiber, S.; Danese, S.; Feagan, B.G.; Reinisch, W.; Niezychowski, W.; et al. Tofacitinib as Induction and Maintenance Therapy for Ulcerative Colitis. N. Engl. J. Med. 2017, 376, 1723–1736. [Google Scholar] [CrossRef]
  59. Vieujean, S.; Peyrin-Biroulet, L. Pharmacokinetics of S1P receptor modulators in the treatment of ulcerative colitis. Expert. Opin. Drug Metab. Toxicol. 2024, 1–12. [Google Scholar] [CrossRef]
  60. Bencardino, S.; D’Amico, F.; Faggiani, I.; Bernardi, F.; Allocca, M.; Furfaro, F.; Parigi, T.L.; Zilli, A.; Fiorino, G.; Peyrin-Biroulet, L.; et al. Efficacy and Safety of S1P1 Receptor Modulator Drugs for Patients with Moderate-to-Severe Ulcerative Colitis. J. Clin. Med. 2023, 12. [Google Scholar] [CrossRef]
  61. Pugliese, D.; Felice, C.; Papa, A.; Gasbarrini, A.; Rapaccini, G.L.; Guidi, L.; Armuzzi, A. Anti TNF-alpha therapy for ulcerative colitis: current status and prospects for the future. Expert. Rev. Clin. Immunol. 2017, 13, 223–233. [Google Scholar] [CrossRef]
  62. Sandborn, W.J.; Baert, F.; Danese, S.; Krznaric, Z.; Kobayashi, T.; Yao, X.; Chen, J.; Rosario, M.; Bhatia, S.; Kisfalvi, K.; et al. Efficacy and Safety of Vedolizumab Subcutaneous Formulation in a Randomized Trial of Patients With Ulcerative Colitis. Gastroenterology 2020, 158, 562–572.e512. [Google Scholar] [CrossRef] [PubMed]
  63. Kucharzik, T.; Koletzko, S.; Kannengiesser, K.; Dignass, A. Ulcerative Colitis-Diagnostic and Therapeutic Algorithms. Dtsch. Arztebl. Int. 2020, 117, 564–574. [Google Scholar] [CrossRef]
  64. Lasa, J.S.; Olivera, P.A.; Danese, S.; Peyrin-Biroulet, L. Efficacy and safety of biologics and small molecule drugs for patients with moderate-to-severe ulcerative colitis: a systematic review and network meta-analysis. Lancet Gastroenterol. Hepatol. 2022, 7, 161–170. [Google Scholar] [CrossRef] [PubMed]
  65. Hirten, R.P.; Sands, B.E. New Therapeutics for Ulcerative Colitis. Annu. Rev. Med. 2021, 72, 199–213. [Google Scholar] [CrossRef] [PubMed]
  66. Burri, E.; Maillard, M.H.; Schoepfer, A.M.; Seibold, F.; Van Assche, G.; Riviere, P.; Laharie, D.; Manz, M.; Swiss Ibdnet, a.o.w.g.o.t.S.S.o.G. Treatment Algorithm for Mild and Moderate-to-Severe Ulcerative Colitis: An Update. Digestion 2020, 101 Suppl. 1, 2–15. [Google Scholar] [CrossRef] [PubMed]
  67. Siegel, R.L.; Miller, K.D.; Wagle, N.S.; Jemal, A. Cancer statistics, 2023. CA Cancer J Clin 2023, 73, 17–48. [Google Scholar] [CrossRef]
  68. Faye, A.S.; Holmer, A.K.; Axelrad, J.E. Cancer in Inflammatory Bowel Disease. Gastroenterol. Clin. North. Am. 2022, 51, 649–666. [Google Scholar] [CrossRef]
  69. Shin, A.E.; Giancotti, F.G.; Rustgi, A.K. Metastatic colorectal cancer: mechanisms and emerging therapeutics. Trends Pharmacol. Sci. 2023, 44, 222–236. [Google Scholar] [CrossRef]
  70. Baxi, V.; Edwards, R.; Montalto, M.; Saha, S. Digital pathology and artificial intelligence in translational medicine and clinical practice. Mod. Pathol. 2022, 35, 23–32. [Google Scholar] [CrossRef]
  71. Carreras, J. Celiac Disease Deep Learning Image Classification Using Convolutional Neural Networks. J. Imaging 2024, 10. [Google Scholar] [CrossRef]
  72. Tsuda, S.; Carreras, J.; Kikuti, Y.Y.; Nakae, H.; Dekiden-Monma, M.; Imai, J.; Tsuruya, K.; Nakamura, J.; Tsukune, Y.; Uchida, T.; et al. Prediction of steroid demand in the treatment of patients with ulcerative colitis by immunohistochemical analysis of the mucosal microenvironment and immune checkpoint: role of macrophages and regulatory markers in disease severity. Pathol. Int. 2019, 69, 260–271. [Google Scholar] [CrossRef] [PubMed]
  73. Mohammed Vashist, N.; Samaan, M.; Mosli, M.H.; Parker, C.E.; MacDonald, J.K.; Nelson, S.A.; Zou, G.Y.; Feagan, B.G.; Khanna, R.; Jairath, V. Endoscopic scoring indices for evaluation of disease activity in ulcerative colitis. Cochrane Database Syst. Rev. 2018, 1, CD011450. [Google Scholar] [CrossRef] [PubMed]
  74. Baron, J.H.; Connell, A.M.; Lennard-Jones, J.E. Variation between Observers in Describing Mucosal Appearances in Proctocolitis. Br. Med. J. 1964, 1, 89–92. [Google Scholar] [CrossRef] [PubMed]
  75. Feagan, B.G.; Greenberg, G.R.; Wild, G.; Fedorak, R.N.; Pare, P.; McDonald, J.W.; Dube, R.; Cohen, A.; Steinhart, A.H.; Landau, S.; et al. Treatment of ulcerative colitis with a humanized antibody to the alpha4beta7 integrin. N. Engl. J. Med. 2005, 352, 2499–2507. [Google Scholar] [CrossRef] [PubMed]
  76. Geboes, K.; Riddell, R.; Ost, A.; Jensfelt, B.; Persson, T.; Lofberg, R. A reproducible grading scale for histological assessment of inflammation in ulcerative colitis. Gut 2000, 47, 404–409. [Google Scholar] [CrossRef]
  77. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” Communications of the ACM 60, no. 6 (May 24, 2017): 84–90. [CrossRef]
  78. Huang, Gao, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. “Densely Connected Convolutional Networks.” In CVPR, vol. 1, no. 2, p. 3. 2017.
  79. Mingxing Tan and Quoc, V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ArXiv Preprint ArXiv:1905.1194, 2019.
  80. Places. http://places2.csail.mit.edu/.
  81. Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. “Going deeper with convolutions.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9. 2015.
  82. Szegedy, Christian, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. “Rethinking the inception architecture for computer vision.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826. 2016.
  83. Sandler, M. , Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C. “MobileNetV2: Inverted Residuals and Linear Bottlenecks.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4510-4520). IEEE.
  84. Zoph, Barret, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. “Learning Transferable Architectures for Scalable Image Recognition.” arXiv preprint arXiv:1707.07012 2, no. 6 (2017).
  85. He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep residual learning for image recognition.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. 2016.
  86. Zhang, Xiangyu, Xinyu Zhou, Mengxiao Lin, and Jian Sun. “ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices.” arXiv preprint arXiv:1707.01083v2 (2017).
  87. Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).
  88. Chollet, F. , 2017. “Xception: Deep Learning with Depthwise Separable Convolutions.” arXiv preprint, pp.1610-02357.
  89. Feakins, R.M. Ulcerative colitis or Crohn’s disease? Pitfalls and problems. Histopathology 2014, 64, 317–335. [Google Scholar] [CrossRef]
  90. Ordas, I.; Eckmann, L.; Talamini, M.; Baumgart, D.C.; Sandborn, W.J. Ulcerative colitis. Lancet 2012, 380, 1606–1619. [Google Scholar] [CrossRef]
  91. Kawachi, H. Histopathological diagnosis of ulcerative colitis-associated neoplasia. Dig. Endosc. 2019, 31 Suppl. 1, 31–35. [Google Scholar] [CrossRef]
  92. Jauregui-Amezaga, A.; Geerits, A.; Das, Y.; Lemmens, B.; Sagaert, X.; Bessissow, T.; Lobaton, T.; Ferrante, M.; Van Assche, G.; Bisschops, R.; et al. A Simplified Geboes Score for Ulcerative Colitis. J. Crohns Colitis 2017, 11, 305–313. [Google Scholar] [CrossRef]
  93. Poulogiannis, G.; Frayling, I.M.; Arends, M.J. DNA mismatch repair deficiency in sporadic colorectal cancer and Lynch syndrome. Histopathology 2010, 56, 167–179. [Google Scholar] [CrossRef]
  94. Cerretelli, G.; Ager, A.; Arends, M.J.; Frayling, I.M. Molecular pathology of Lynch syndrome. J. Pathol. 2020, 250, 518–531. [Google Scholar] [CrossRef] [PubMed]
  95. Landskron, G.; De la Fuente Lopez, M.; Dubois-Camacho, K.; Diaz-Jimenez, D.; Orellana-Serradell, O.; Romero, D.; Sepulveda, S.A.; Salazar, C.; Parada-Venegas, D.; Quera, R.; et al. Interleukin 33/ST2 Axis Components Are Associated to Desmoplasia, a Metastasis-Related Factor in Colorectal Cancer. Front. Immunol. 2019, 10, 1394. [Google Scholar] [CrossRef]
  96. Nagtegaal, I.D.; Odze, R.D.; Klimstra, D.; Paradis, V.; Rugge, M.; Schirmacher, P.; Washington, K.M.; Carneiro, F.; Cree, I.A.; Board, W.H.O.C.o.T.E. The 2019 WHO classification of tumours of the digestive system. Histopathology 2020, 76, 182–188. [Google Scholar] [CrossRef] [PubMed]
  97. Ahadi, M.; Sokolova, A.; Brown, I.; Chou, A.; Gill, A.J. The 2019 World Health Organization Classification of appendiceal, colorectal and anal canal tumours: an update and critical assessment. Pathology 2021, 53, 454–461. [Google Scholar] [CrossRef]
  98. Dunne, P.D.; Arends, M.J. Molecular pathological classification of colorectal cancer-an update. Virchows Arch. 2024, 484, 273–285. [Google Scholar] [CrossRef]
  99. Muller, M.F.; Ibrahim, A.E.; Arends, M.J. Molecular pathological classification of colorectal cancer. Virchows Arch. 2016, 469, 125–134. [Google Scholar] [CrossRef]
  100. O’Connell, J.B.; Maggard, M.A.; Livingston, E.H.; Yo, C.K. Colorectal cancer in the young. Am. J. Surg. 2004, 187, 343–348. [Google Scholar] [CrossRef]
  101. Kather, J.N.; Weis, C.A.; Bianconi, F.; Melchers, S.M.; Schad, L.R.; Gaiser, T.; Marx, A.; Zollner, F.G. Multi-class texture analysis in colorectal cancer histology. Sci. Rep. 2016, 6, 27988. [Google Scholar] [CrossRef] [PubMed]
  102. Treanor, D.; Quirke, P. Pathology of colorectal cancer. Clin Oncol (R Coll Radiol) 2007, 19, 769–776. [Google Scholar] [CrossRef]
  103. Histology For Pathologists. ISBN 1496398947, 978-1496398949.
  104. Kierszenbaum, L. Tres. Histology and Cell Biology: An Introduction to Pathology 5th Edition 2020. May 27, 2020. Elsevier Language: English ISBN-13: 978-0323673211 ISBN-10: 032367321X.
  105. Angkeow, J.; Rothman, A.; Chaaban, L.; Paul, N.; Melia, J. Systematic Review: Outcome Prediction in Acute Severe Ulcerative Colitis. Gastro Hep Adv. 2024, 3, 260–270. [Google Scholar] [CrossRef]
  106. Pabla, B.S.; Schwartz, D.A. Assessing Severity of Disease in Patients with Ulcerative Colitis. Gastroenterol. Clin. North. Am. 2020, 49, 671–688. [Google Scholar] [CrossRef] [PubMed]
  107. Heinz, M.V.; Bhattacharya, S.; Trudeau, B.; Quist, R.; Song, S.H.; Lee, C.M.; Jacobson, N.C. Testing domain knowledge and risk of bias of a large-scale general artificial intelligence model in mental health. Digit. Health 2023, 9, 20552076231170499. [Google Scholar] [CrossRef]
  108. Li, H.; Ma, C.; Chen, X.; Liu, X. Dynamic Consolidation for Continual Learning. Neural Comput. 2023, 35, 228–248. [Google Scholar] [CrossRef] [PubMed]
  109. Oliveira, O.N., Jr.; Christino, L.; Oliveira, M.C.F.; Paulovich, F.V. Artificial Intelligence Agents for Materials Sciences. J. Chem. Inf. Model. 2023, 63, 7605–7609. [Google Scholar] [CrossRef]
  110. Korteling, J.E.H.; van de Boer-Visschedijk, G.C.; Blankendaal, R.A.M.; Boonekamp, R.C.; Eikelboom, A.R. Human- versus Artificial Intelligence. Front. Artif. Intell. 2021, 4, 622364. [Google Scholar] [CrossRef]
  111. Stanley, K. Artificial Intelligence and the Future of Dentistry. Compend. Contin. Educ. Dent. 2023, 44, 250–253. [Google Scholar]
  112. Artificial Intelligence. IBM Design for AI. Available online: https://www.ibm.com/design/ai/basics/ai/ (accessed on 26 September 2024).
  113. Bharadwaj, S.; Narula, N.; Tandon, P.; Yaghoobi, M. Role of endoscopy in inflammatory bowel disease. Gastroenterol Rep (Oxf) 2018, 6, 75–82. [Google Scholar] [CrossRef] [PubMed]
  114. Gajendran, M.; Loganathan, P.; Jimenez, G.; Catinella, A.P.; Ng, N.; Umapathy, C.; Ziade, N.; Hashash, J.G. A comprehensive review and update on ulcerative colitis(). Dis. Mon. 2019, 65, 100851. [Google Scholar] [CrossRef]
  115. Computer vision. MathWorks. R2024b. Available online: https://mathworks.com (accessed on 26 September 2024).
  116. van der Velden, B.H.M.; Kuijf, H.J.; Gilhuijs, K.G.A.; Viergever, M.A. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Med. Image Anal. 2022, 79, 102470. [Google Scholar] [CrossRef]
  117. Tosun, A.B.; Pullara, F.; Becich, M.J.; Taylor, D.L.; Fine, J.L.; Chennubhotla, S.C. Explainable AI (xAI) for Anatomic Pathology. Adv. Anat. Pathol. 2020, 27, 241–250. [Google Scholar] [CrossRef]
  118. de Vries, B.M.; Zwezerijnen, G.J.C.; Burchell, G.L.; van Velden, F.H.P.; Menke-van der Houven van Oordt, C.W.; Boellaard, R. Explainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review. Front Med (Lausanne) 2023, 10, 1180773. [Google Scholar] [CrossRef] [PubMed]
  119. O’Sullivan, S.; Janssen, M.; Holzinger, A.; Nevejans, N.; Eminaga, O.; Meyer, C.P.; Miernik, A. Explainable artificial intelligence (XAI): closing the gap between image analysis and navigation in complex invasive diagnostic procedures. World J. Urol. 2022, 40, 1125–1134. [Google Scholar] [CrossRef] [PubMed]
  120. Ramprasaath, R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. [CrossRef]
  121. Selvaraju, R.R., M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.” In IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618–626.
  122. Arun Das, Paul Rad. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv:2006.11371.
  123. Turan, M.; Durmus, F. UC-NfNet: Deep learning-enabled assessment of ulcerative colitis from colonoscopy images. Med. Image Anal. 2022, 82, 102587. [Google Scholar] [CrossRef]
  124. Jiang, X.; Luo, X.; Nan, Q.; Ye, Y.; Miao, Y.; Miao, J. Application of deep learning in the diagnosis and evaluation of ulcerative colitis disease severity. Therap Adv. Gastroenterol. 2023, 16, 17562848231215579. [Google Scholar] [CrossRef]
  125. Bhambhvani, H.P.; Zamora, A. Deep learning enabled classification of Mayo endoscopic subscore in patients with ulcerative colitis. Eur. J. Gastroenterol. Hepatol. 2021, 33, 645–649. [Google Scholar] [CrossRef]
  126. Fan, Y.; Mu, R.; Xu, H.; Xie, C.; Zhang, Y.; Liu, L.; Wang, L.; Shi, H.; Hu, Y.; Ren, J.; et al. Novel deep learning-based computer-aided diagnosis system for predicting inflammatory activity in ulcerative colitis. Gastrointest. Endosc. 2023, 97, 335–346. [Google Scholar] [CrossRef] [PubMed]
  127. Byrne, M.F.; Panaccione, R.; East, J.E.; Iacucci, M.; Parsa, N.; Kalapala, R.; Reddy, D.N.; Ramesh Rughwani, H.; Singh, A.P.; Berry, S.K.; et al. Application of Deep Learning Models to Improve Ulcerative Colitis Endoscopic Disease Activity Scoring Under Multiple Scoring Systems. J. Crohns Colitis 2023, 17, 463–471. [Google Scholar] [CrossRef]
  128. Qi, J.; Ruan, G.; Ping, Y.; Xiao, Z.; Liu, K.; Cheng, Y.; Liu, R.; Zhang, B.; Zhi, M.; Chen, J.; et al. Development and validation of a deep learning-based approach to predict the Mayo endoscopic score of ulcerative colitis. Therap Adv. Gastroenterol. 2023, 16, 17562848231170945. [Google Scholar] [CrossRef]
  129. Gutierrez Becker, B.; Arcadu, F.; Thalhammer, A.; Gamez Serna, C.; Feehan, O.; Drawnel, F.; Oh, Y.S.; Prunotto, M. Training and deploying a deep learning model for endoscopic severity grading in ulcerative colitis using multicenter clinical trial data. Ther. Adv. Gastrointest. Endosc. 2021, 14, 2631774521990623. [Google Scholar] [CrossRef]
  130. Kim, J.E.; Choi, Y.H.; Lee, Y.C.; Seong, G.; Song, J.H.; Kim, T.J.; Kim, E.R.; Hong, S.N.; Chang, D.K.; Kim, Y.H.; et al. Deep learning model for distinguishing Mayo endoscopic subscore 0 and 1 in patients with ulcerative colitis. Sci. Rep. 2023, 13, 11351. [Google Scholar] [CrossRef]
  131. Rymarczyk, D.; Schultz, W.; Borowa, A.; Friedman, J.R.; Danel, T.; Branigan, P.; Chalupczak, M.; Bracha, A.; Krawiec, T.; Warchol, M.; et al. Deep Learning Models Capture Histological Disease Activity in Crohn’s Disease and Ulcerative Colitis with High Fidelity. J. Crohns Colitis 2024, 18, 604–614. [Google Scholar] [CrossRef] [PubMed]
  132. Vande Casteele, N.; Leighton, J.A.; Pasha, S.F.; Cusimano, F.; Mookhoek, A.; Hagen, C.E.; Rosty, C.; Pai, R.K.; Pai, R.K. Utilizing Deep Learning to Analyze Whole Slide Images of Colonic Biopsies for Associations Between Eosinophil Density and Clinicopathologic Features in Active Ulcerative Colitis. Inflamm. Bowel Dis. 2022, 28, 539–546. [Google Scholar] [CrossRef] [PubMed]
  133. Ohara, J.; Nemoto, T.; Maeda, Y.; Ogata, N.; Kudo, S.E.; Yamochi, T. Deep learning-based automated quantification of goblet cell mucus using histological images as a predictor of clinical relapse of ulcerative colitis with endoscopic remission. J. Gastroenterol. 2022, 57, 962–970. [Google Scholar] [CrossRef]
  134. Peyrin-Biroulet, L.; Adsul, S.; Stancati, A.; Dehmeshki, J.; Kubassova, O. An artificial intelligence-driven scoring system to measure histological disease activity in ulcerative colitis. United European Gastroenterol. J. 2024. [Google Scholar] [CrossRef]
  135. Helou, D.G.; Quach, C.; Hurrell, B.P.; Li, X.; Li, M.; Akbari, A.; Shen, S.; Shafiei-Jahani, P.; Akbari, O. LAIR-1 limits macrophage activation in acute inflammatory lung injury. Mucosal Immunol. 2023, 16, 788–800. [Google Scholar] [CrossRef]
  136. Carreras, J.; Ikoma, H.; Kikuti, Y.Y.; Miyaoka, M.; Hiraiwa, S.; Tomita, S.; Kondo, Y.; Ito, A.; Nagase, S.; Miura, H.; et al. Mutational, immune microenvironment, and clinicopathological profiles of diffuse large B-cell lymphoma and follicular lymphoma with BCL6 rearrangement. Virchows Arch. 2024, 484, 657–676. [Google Scholar] [CrossRef] [PubMed]
  137. Hassan-Zahraee, M.; Ye, Z.; Xi, L.; Dushin, E.; Lee, J.; Romatowski, J.; Leszczyszyn, J.; Danese, S.; Sandborn, W.J.; Banfield, C.; et al. Baseline Serum and Stool Microbiome Biomarkers Predict Clinical Efficacy and Tissue Molecular Response After Ritlecitinib Induction Therapy in Ulcerative Colitis. J. Crohns Colitis 2024, 18, 1361–1370. [Google Scholar] [CrossRef]
  138. Seo, H.; Chen, J.; Gonzalez-Avalos, E.; Samaniego-Castruita, D.; Das, A.; Wang, Y.H.; Lopez-Moyado, I.F.; Georges, R.O.; Zhang, W.; Onodera, A.; et al. TOX and TOX2 transcription factors cooperate with NR4A transcription factors to impose CD8(+) T cell exhaustion. Proc. Natl. Acad. Sci. U S A 2019, 116, 12410–12415. [Google Scholar] [CrossRef]
  139. Blank, C.U.; Haining, W.N.; Held, W.; Hogan, P.G.; Kallies, A.; Lugli, E.; Lynn, R.C.; Philip, M.; Rao, A.; Restifo, N.P.; et al. Defining ‘T cell exhaustion’. Nat. Rev. Immunol. 2019, 19, 665–674. [Google Scholar] [CrossRef]
  140. Canaria, D.A.; Rodriguez, J.A.; Wang, L.; Yeo, F.J.; Yan, B.; Wang, M.; Campbell, C.; Kazemian, M.; Olson, M.R. Tox induces T cell IL-10 production in a BATF-dependent manner. Front. Immunol. 2023, 14, 1275423. [Google Scholar] [CrossRef]
  141. Gu, X.G.; Yu, X.; Zhou, B.Y.; Li, M.; Xu, W.; Li, Y.; Li, L.F. Immune Cell Profiling of Atopic Dermatitis Patients Before and After Treatment with Halometasone Cream Wet-Wrap Therapy by Single-Cell Sequencing. Indian. J. Dermatol. 2023, 68, 8–14. [Google Scholar] [CrossRef] [PubMed]
Preprints 138358 g005
Figure 5. Network performance during training and validation. The data were partitioned into a training set (70% of the images) to train the network and a validation set (10%) to test the performance of the network during the training; and a test set (20%) was used as a holdout to test the performance of the trained network on new data. This figure shows the accuracy and loss during the training (70%) and validation (10%) sets. The CNN was based on transfer learning from ResNet-18).
Figure 5. Network performance during training and validation. The data were partitioned into a training set (70% of the images) to train the network and a validation set (10%) to test the performance of the network during the training; and a test set (20%) was used as a holdout to test the performance of the trained network on new data. This figure shows the accuracy and loss during the training (70%) and validation (10%) sets. The CNN was based on transfer learning from ResNet-18).
Preprints 138358 g005
Preprints 138358 g006
Figure 6. Confusion matrix of the test dataset (new data). The data were partitioned into a training set (70% of the images) to train the network and a validation set (10%) to test the performance of the network during the training; and a test set (20%) was used as a holdout to test the performance of the trained network on new data. ADK, adenocarcinoma (colorectal cancer); CC, colon control; UC, ulcerative colitis.
Figure 6. Confusion matrix of the test dataset (new data). The data were partitioned into a training set (70% of the images) to train the network and a validation set (10%) to test the performance of the network during the training; and a test set (20%) was used as a holdout to test the performance of the trained network on new data. ADK, adenocarcinoma (colorectal cancer); CC, colon control; UC, ulcerative colitis.
Preprints 138358 g006
Preprints 138358 g007
Figure 7. Explanation of network predictions using Grad-CAM. Grad-CAM was used to visualize which parts of the image were important for the classification decision (diagnosis) of the network. The most relevant parts are highlighted in red (jet colormap). The prediction scores for each diagnosis are shown below each hematoxylin and eosin (H&E) image.
Figure 7. Explanation of network predictions using Grad-CAM. Grad-CAM was used to visualize which parts of the image were important for the classification decision (diagnosis) of the network. The most relevant parts are highlighted in red (jet colormap). The prediction scores for each diagnosis are shown below each hematoxylin and eosin (H&E) image.
Preprints 138358 g007
Preprints 138358 g008
Figure 8. Grad-CAM analysis of incorrectly classified images. Grad-CAM analysis was used to visualize which parts of the image were important for the classification decision (diagnosis) of the network. In most cases, the classification error was due to a wrong focalized area in the image by the neural network or the image was not diagnostic from a histopathological point of view.
Figure 8. Grad-CAM analysis of incorrectly classified images. Grad-CAM analysis was used to visualize which parts of the image were important for the classification decision (diagnosis) of the network. In most cases, the classification error was due to a wrong focalized area in the image by the neural network or the image was not diagnostic from a histopathological point of view.
Preprints 138358 g008
Preprints 138358 g009
Figure 9. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 79.53.
Figure 9. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 79.53.
Preprints 138358 g009
Preprints 138358 g010
Figure 10. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive using LAIR1 immunohistochemistry. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 88.31%.
Figure 10. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive using LAIR1 immunohistochemistry. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 88.31%.
Preprints 138358 g010
Preprints 138358 g011
Figure 11. LAIR1 immunohistochemistry in the ulcerative colitis dataset, and classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative colitis cases. Overall, the inflammatory component was higher in steroid-requiring cases.
Figure 11. LAIR1 immunohistochemistry in the ulcerative colitis dataset, and classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative colitis cases. Overall, the inflammatory component was higher in steroid-requiring cases.
Preprints 138358 g011
Preprints 138358 g012
Figure 12. Examples of cropped images of LAIR1 immunohistochemistry in the ulcerative colitis dataset. The cropped images were used as input data in the CNN, which managed to classify between steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative cases. Overall, the inflammatory component was higher in steroid-requiring cases.
Figure 12. Examples of cropped images of LAIR1 immunohistochemistry in the ulcerative colitis dataset. The cropped images were used as input data in the CNN, which managed to classify between steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative cases. Overall, the inflammatory component was higher in steroid-requiring cases.
Preprints 138358 g012
Preprints 138358 g013
Figure 13. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive using TOX2 immunohistochemistry. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 85.62%.
Figure 13. Confusion matrix of the test dataset for the classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive using TOX2 immunohistochemistry. The test dataset included new data (holdout, 20%). The analysis was based on transfer learning from ResNet-18. The accuracy was 85.62%.
Preprints 138358 g013
Preprints 138358 g014
Figure 14. TOX2 immunohistochemistry in the ulcerative colitis dataset, and classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative colitis cases. Overall, the TOX2-positive inflammatory component was higher in mesalazine-responsive cases.
Figure 14. TOX2 immunohistochemistry in the ulcerative colitis dataset, and classification of steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative colitis cases. Overall, the TOX2-positive inflammatory component was higher in mesalazine-responsive cases.
Preprints 138358 g014
Preprints 138358 g015
Figure 15. Examples of cropped images of TOX2 immunohistochemistry in the ulcerative colitis dataset. The cropped images were used as input data in the CNN, which managed to classify between steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative cases. Overall, the TOX2-positive inflammatory component was higher in mesalazine-responsive cases.
Figure 15. Examples of cropped images of TOX2 immunohistochemistry in the ulcerative colitis dataset. The cropped images were used as input data in the CNN, which managed to classify between steroid-requiring and nonsteroid-requiring/mesalazine-responsive ulcerative cases. Overall, the TOX2-positive inflammatory component was higher in mesalazine-responsive cases.
Preprints 138358 g015
Table 1. Pathogenesis of inflammatory bowel disease.
Table 1. Pathogenesis of inflammatory bowel disease.
Mechanisms Key players
Dysregulation of the epithelial barrier Alterations of the mucus, increased number of bacteria within the mucus, and increased intestinal permeability [6,7,8]
Dysregulation of immune cells Increased recruitment and activation of immune cell, including myeloid inflammatory cells, natural killer cells, T cells, B cells, plasma cells, neutrophils, and other leukocytes [9,10,11,12,13,14,15,16,17].
Dysregulation of immune regulators and inflammatory cytokines CD4+T lymphocytes, interferon (IFN)-gamma, Th1, Th2, Th17, FOXP3+regulatory T lymphocytes (Tregs), IL-10, TGFB, CD8+cytotoxic T lymphocytes [18,19,20,21,22,23,24,25]
Microbes Alterations in the diversity and density of bacteria [26,27,28,29], specific microbial components, intestinal viruses [9,30,31,32], and fungi [33,34,35]
Genetic susceptibility Over 240 different susceptibility loci, NOD2, ATG16L1, NADPH, and immune-related (Th17/IL-23, IL-10, TNFSF15, cytokine, adaptive immune response and epithelial pathways) [14,36,37,38,39,40,41,42,43,44]
Table 2. Clinicopathological characteristics of ulcerative colitis.
Table 2. Clinicopathological characteristics of ulcerative colitis.
Variable
No. (%)		 Mesalazine
-responsive		 Steroid
-requiring		 All cases P value
Number of patients 22 13 35
Age (mean ±STD) 43.7 ±13.6 29.5 ±17.6 38.4 ±16.4 0.012
Sex Male 14/22 (63.6) 6/13 (46.2) 20/35 (57.1) 0.481
Colon biopsy location
Ascending 0/22 (0) 1/13 (7.7) 1/35 (2.9) 0.009
Transverse 0/22 (0) 2/13 (15.4) 2/35 (5.7)
Descending 2/22 (9.1) 3/13 (23.1) 5/35 (14.3)
Sigmoid 2/22 (9.1) 3/13 (23.1) 5/35 (14.3)
Rectum 18/22 (81.8) 4/13 (30.8) 22/35 (62.9)
Endoscopic Baron score
1 13/22 (59.1) 2/13 (15.4 15/35 (42.9) 0.009
2 9/22 (40.9) 8/13 (61.5) 17/35 (48.6)
3 0/22 (0) 3/13 (23.1) 3/35 (8.6)
Histologic Geboes score
1 2/22 (9.1) 0/13 (0) 2/35 (5.) 0.101 (0.007*)
2 13/22 (59.1) 4/13 (30.8) 17/35 (48.6)
3 5/22 (22.7) 3/13 (23.1) 8/35 (22.9)
4 2/22 (9.1) 5/13 (38.5) 7/35 (20)
5 0/22 (0) 1/13 (7.7) 1/35 (2.9)
*Linear-by-linear association within Chi-Square test.
Table 3. Design and training parameters of the convolutional neural network.
Table 3. Design and training parameters of the convolutional neural network.
ResNet-18-based CNN Training (70%) Validation (10%) Training options
Input type: image
Output type: classification
Number of layers: 71
Number of connections: 78		 Observations: 59677
Classes: 3
Ulcerative colitis: 6497
Colorectal cancer: 44608
Colon control: 8572		 Observations: 8525
Classes: 3
Ulcerative colitis: 928
Colorectal cancer: 6372
Colon control: 1225		 Solver: sgdm
Initial learning rate: 0.001
MiniBatch size: 128
MaxEpochs: 5
Validation frequency: 50
Iterations: 2330
Iterations per epoch: 466
Based on ResNet-18 transfer learning. Convolutional neural network (CNN).
Table 4. Pretrained Neural Networks.
Table 4. Pretrained Neural Networks.
Name [Reference] Model Name Argument Depth Size (MB) Parameters (Millions) Image Input Size
AlexNet [77] “alexnet” 8 227 61 227-by-227
DenseNet-201 [78] “densenet201” 201 77 20 224-by-224
EfficientNet-b0 [79] “efficientnetb0” 82 20 5.3 224-by-224
GoogLeNet [80,81] “googlenet” 22 27 7 224-by-224
“googlenet-places365”
Inception-v3 [82] “inceptionv3” 48 89 23.9 299-by-299
MobileNet-v2 [83] “mobilenetv2” 53 13 3.5 224-by-224
NASNet-Large [84] “nasnetlarge” * 332 88.9 331-by-331
NASNet-Mobile [84] “nasnetmobile” * 20 5.3 224-by-224
ResNet-18 [85] “resnet18” 18 44 11.7 224-by-224
ResNet-50 [85] “resnet50” 50 96 25.6 224-by-224
ResNet-101 [85] “resnet101” 101 167 44.6 224-by-224
ShuffleNet [86] “shufflenet” 50 5.4 1.4 224-by-224
VGG-16 [87] “vgg16” 16 515 138 224-by-224
VGG-19 [87] “vgg19” 19 535 144 224-by-224
Xception [88] “xception” 71 85 22.9 299-by-299
Table 5. Parameters of network performance of the test dataset (holdout, new data).
Table 5. Parameters of network performance of the test dataset (holdout, new data).
Predicted variable Accuracy (%) Precision (%) Recall (%) F1-Score (%) Specificity (%) False Positive Rate (%)
Ulcerative colitis 99.10 97.09 94.79 95.93 99.64 0.36
Adenocarcinoma 99.84 99.90 99.88 99.89 99.70 0.30
Colon control 99.06 95.75 97.63 96.68 99.29 0.71
Based on ResNet-18 transfer learning. Recall also refers to sensitivity and the true positive rate (TPR). False positive rate (FPR).
Table 6. Parameters of network performance of the test dataset (holdout, new data) using H&E images.
Table 6. Parameters of network performance of the test dataset (holdout, new data) using H&E images.
Predicted variable Accuracy (%) Precision (%) Recall (%) F1-Score (%) Specificity (%) False Positive Rate (%)
Steroid-requiring 79.53 66.18 70.74 68.39 83.53 16.47
Mesalazine-responsive 79.53 86.23 83.53 84.86 70.74 29.26
Based on ResNet-18 transfer learning. Recall also refers to sensitivity and the true positive rate (TPR). False positive rate (FPR).
Table 7. Parameters of network performance of the test dataset (holdout, new data) using LAIR1 immunohistochemistry.
Table 7. Parameters of network performance of the test dataset (holdout, new data) using LAIR1 immunohistochemistry.
Predicted variable Accuracy (%) Precision (%) Recall (%) F1-Score (%) Specificity (%) False Positive Rate (%)
Steroid-requiring 88.31 79.38 82.26 80.79 90.89 9.11
Mesalazine-responsive 88.31 92.31 90.89 91.59 82.26 17.74
Based on ResNet-18 transfer learning. Recall also refers to sensitivity and the true positive rate (TPR). False positive rate (FPR).
Table 8. Parameters of network performance of the test dataset (holdout, new data) using TOX2 immunohistochemistry.
Table 8. Parameters of network performance of the test dataset (holdout, new data) using TOX2 immunohistochemistry.
Predicted variable Accuracy (%) Precision (%) Recall (%) F1-Score (%) Specificity (%) False Positive Rate (%)
Steroid-requiring 85.62 72.04 79.51 75.59 87.99 12.01
Mesalazine-responsive 85.62 91.69 87.99 89.80 79.51 20.49
Based on ResNet-18 transfer learning. Recall also refers to sensitivity and the true positive rate (TPR). False positive rate (FPR).
Table 9. Performance comparison of CNN networks on the test set (holdout, new data) using H&E images.
Table 9. Performance comparison of CNN networks on the test set (holdout, new data) using H&E images.
Model Accuracy (%)
DenseNet-201 99.30
ResNet-50 99.14
Inception-v3 99.13
ResNet-101 99.10
ResNet-18 99.00
ShuffleNet 98.94
MobileNet-v2 98.89
NasNet-Large 98.88
GoogLeNet-Places365 98.86
VGG-19 98.80
EfficientNet-b0 98.79
AlexNet 98.77
Xception 98.66
VGG-16 98.65
GoogLeNet 98.60
NasNet-Mobile 98.58
This analysis was performed using transfer learning from several types of pretrained convolutional neural networks (CNN) for image classification. Transfer learning using several pretrained CNN was used to classify images of ulcerative colitis, colorectal cancer (adenocarcinoma), and colon control.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated