Convolution Neural Network Based Multi- Label Disease Detection using Tongue Images

1. Introduction

Technological breakthrough has made clinical investigations and test for disease diagnosis feasible to great extent. Despite of available high end diagnostic tools some common ailments can be diagnosed by observation of certain human indices such as pulse (nadi), eyes, nails and tongue as per ayurveda. Ancient traditional medicine made these parameters as means of diagnosis with no advanced technology tools at hand to predict many abnormalities related to the health status of internal body organs. The paper presents a novel method for multilabel disease classification from tongue analysis.

The use of decision support systems has been increasing over the past decades, modern deep learning models allow reliable classification and object detection of medical images showing remarkable accuracy comparable to that of physicians. An automated tongue diagnosis system is a means to bridge the gap between traditional method of diagnosis and modern western medicine. Subjective nature of the same related to the knowledge base and thorough practice of an expert traditional practitioner can be vanquished using quantitative analysis to make it accountable and acceptable. In this paper we have developed a multi disease classification algorithm which is able to predict multiple disease diagnosis with appreciable accuracy. Eight diseased states are considered based on the collected data samples. The Contributions of this paper are as follows:

• This paper presents a MobileNetV2 deep learning model to detect the disease tongue from a normal healthy tongue. Five-fold cross validation and transfer learning technique is used for this binary classification. Main achievement is sanguine results achieved with images taken by smartphone cameras rather than by standard equipment.
• Multilabel classification model is proposed for eight common categories of ailments. DenseNet 121 architecture is utilized for disease classification, satisfactory results achieved with the small dataset. Eight disease labels are diabetes, hypertension, acidic peptic disease, pyrexia, hepatitis, cold cough, gastritis and others.

Organization of this paper is as follows: ‘Related Work’ summarizes the tongue feature extraction techniques used. ‘Methodology’ section elaborates upon flow of the related work, dataset and training details along with experimentation and evaluation Metrics used, and finally performance analysis is reported in ‘Results and Discussion’ section. Conclusion and Future Work’ section finally concludes are research work.

2. Related Work

Quantifying the tongue diagnostic attributes is one of the main challenges in automation of tongue analysis for disease diagnosis. Study of tongue conditions based on ayurveda are presented by [1] related to health status of an individual. Summary different tongue attributes like colour coating, texture and geometric shape are analysed for predicting any particular diseased condition by [2] as practiced in oriental medicine practices. Automation of tongue analysis system essentially requires an image capturing device with high resolution images for accurate extraction of tongue features for disease predictions in agreement with the traditional medicine practitioners. Different imaging setups are explored using high end CCD cameras [3,4,5,6,7,8,9], Hyperspectral cameras [10,11,12,13,14,15,16] to smartphone cameras [17,18,19,20,21,22,23,24]. Tongue region needs to be segmented from the images captured which consist of teeth lips skin area before feature extraction and classification. Conventional approaches as well as ANN and deep learning algorithms have been explored over a period of time for tongue area segmentation and feature extraction. Most of the work done in this area targeted a particular disease and tongue features related to it, probably due to lack of digital dataset pertaining to all features possible for various diseases. Some common diseases such as diabetes, appendicitis, gastritis was targeted and relevant features and classification was done using statistical techniques [25,26,27,28]. Hybrid model with statistical methods for feature extraction and machine learning algorithms for classification were also developed [29,30,31,32,33].

Table 1. Summary of literature review of ML based models for specific tongue feature.

Tongue Features / Disease targeted	Dataset	Method employed / Disease investigated	Reference
5 tongue body colour, 6 tongue coating colour	1080 subjects	k-means clustering algorithm applied to the images acquired using DSO1 state of the art acquisition system	[34]
Diabetes	732 subjects	TFDA-1 used to capture images and extract, texture, coating features along with an Auto-Encoder algorithm to extract tongue features then fusion of the two set of features done using k-means algorithm for classification	[35]
Tongue area detection calibration and constitution classification	50 subjects	Tongue detection using faster-RCNN, Feature extraction models ResNet-50, VGG-16 and Inception-V3, alongside LBP for texture features and Colour-Moment for colour feature. Model evaluated using classifiers SVM and Decision tree.	[36]
Colour & Texture features	702 images	Gray Level co-occurrence Matrix (GLCM), along with LEAD (Multilabel Learning Algorithm) with threshold determining algorithm for improved results over other existing techniques	[37]
Multifeature extraction	268 images	GLA (Generalized Lloyd Algorithm) to extract colour and texture features from the tongue surface.	[38]
Seven categories fissured tongue, tooth-marked, statis, spotted, greasy, peeled and rotten coating	8676 images	faster R-CNN a region-based network achieved an accuracy of 90.67%	[39]
11 features on the tongue surface considered	482 images	ResNet-34 architecture, 86% accuracy for 11 features identified	[40]
Tooth marked tongue related to spleen defeciency	1548 images	ResNet-34 architecture, 90% accuracy	[41]
Gastritis	263 gastritis patients, 48 healthy.	features related to gastritis were extracted using constrained high dispersal neural network Ada Boost, SVM (support Vector Machine), MLP (Multilayer Perceptron Classifier)	[42] [43]
11 disease categories plus healthy tongue images	936 images, 78 images for each of the 12 disease categories including healthy	extracted tongue features by VGG 19 network supported by Random Forest classifier achieved 93.7% accuracy.	[44]
12 disease categories including healthy	936 images, 78 images for each of the 12 disease categories	Designed IoT base Automated synergic deep learning tongue colour image analysis model giving 98.3% accuracy for disease diagnosis and classification.	[45]
Iron deficiency	95 images from Harvard dataset	Explored the possibility of monitoring health status by tongue images using CNN algorithm which could be deployed on android mobile app.	[46]

Table 1. Summary of literature review of ML based models for specific tongue feature.

Tongue Features / Disease targeted	Dataset	Method employed / Disease investigated	Reference
5 tongue body colour, 6 tongue coating colour	1080 subjects	k-means clustering algorithm applied to the images acquired using DSO1 state of the art acquisition system	[34]
Diabetes	732 subjects	TFDA-1 used to capture images and extract, texture, coating features along with an Auto-Encoder algorithm to extract tongue features then fusion of the two set of features done using k-means algorithm for classification	[35]
Tongue area detection calibration and constitution classification	50 subjects	Tongue detection using faster-RCNN, Feature extraction models ResNet-50, VGG-16 and Inception-V3, alongside LBP for texture features and Colour-Moment for colour feature. Model evaluated using classifiers SVM and Decision tree.	[36]
Colour & Texture features	702 images	Gray Level co-occurrence Matrix (GLCM), along with LEAD (Multilabel Learning Algorithm) with threshold determining algorithm for improved results over other existing techniques	[37]
Multifeature extraction	268 images	GLA (Generalized Lloyd Algorithm) to extract colour and texture features from the tongue surface.	[38]
Seven categories fissured tongue, tooth-marked, statis, spotted, greasy, peeled and rotten coating	8676 images	faster R-CNN a region-based network achieved an accuracy of 90.67%	[39]
11 features on the tongue surface considered	482 images	ResNet-34 architecture, 86% accuracy for 11 features identified	[40]
Tooth marked tongue related to spleen defeciency	1548 images	ResNet-34 architecture, 90% accuracy	[41]
Gastritis	263 gastritis patients, 48 healthy.	features related to gastritis were extracted using constrained high dispersal neural network Ada Boost, SVM (support Vector Machine), MLP (Multilayer Perceptron Classifier)	[42] [43]
11 disease categories plus healthy tongue images	936 images, 78 images for each of the 12 disease categories including healthy	extracted tongue features by VGG 19 network supported by Random Forest classifier achieved 93.7% accuracy.	[44]
12 disease categories including healthy	936 images, 78 images for each of the 12 disease categories	Designed IoT base Automated synergic deep learning tongue colour image analysis model giving 98.3% accuracy for disease diagnosis and classification.	[45]
Iron deficiency	95 images from Harvard dataset	Explored the possibility of monitoring health status by tongue images using CNN algorithm which could be deployed on android mobile app.	[46]

[47] Reviewed current trends in tongue diagnosis. They also trained classification model Random Forest and Support vector machine on tongue dataset with tongue region divided into 5 parts as per internal organs layout, and extracted seven colour spaces from the five extracted parts. Further they trained VGG and ResNet pretrained models on tongue constitution classification. ResNet 50 showed best performance with 64.52 % accuracy. Tremendous efforts have been put in to effectively utilize the full potential of the tongue diagnosis system for over two decades. Enhancement in tools and techniques for achieving the wholistic approach and to overcome the subjective nature of diagnosis is an ongoing process a need to set some standard protocols for it to be readily acceptable by the end users. Majority of the research has been targeted towards a particular disease and worked towards extracting tongue features for the same. Aim of this paper is a step towards developing a decision support system with use of deep architecture to predict a wholistic tongue diagnosis, not just confined to any particular tongue features or diseases.

3. Methodology

The implemented tongue image classification pipeline illustrated in Figure 1, after image acquisition preprocessing and tongue area segmentation, steps involved for classification are be summarized as follows:

• Stratified 5-fold cross validation for diseased risk classification.
• Ensemble learning strategy: bagging, to reduce variance in data.
• Up sampling of dataset to set some minimum sample size in each class.
• Extensive real time data augmentation for training models.
• Class weighted focal loss, to tackle class imbalance.
• Individual training for multi disease classification and disease risk detection.

Figure 1. Proposed pipeline for multi-label disease detection using tongue imaged.

Figure 1. Proposed pipeline for multi-label disease detection using tongue imaged.

3.1. Tongue Analysis Dataset

Total 1095 images of subjects suffering from one or more than one ailment is acquired using smartphone cameras with image resolution greater than equal to 8 mega pixels (Samsung A50, iPhone, one plus). Total 822 images for healthy individuals are also collected through willing individuals. All images are collected after necessary consent from individuals eager to be part of our study. Raw images are of different resolution and various sizes. For the proposed model images are annotated with eight conditions other than normal and disease risk categories as listed in Table 2. Label class others includes some uncommon ones like CAD (coronary artery disease), CKD (chronic kidney disease), COPD (Chronic Obstructive Pulmonary Disease), Epilepsy, Vertigo.

3.2. Preprocessing and Image Augmentation

Images captured are of different sizes and formats primary processing is done by converting all images to jpeg of 256 x 256 pixels. Next step involves segmentation of the tongue area of interest by Double U-Net architecture [48]. In order to increase data variability further preprocessing methods used such as image augmentation for up sampling to balance class distribution and real time augmentation during training to obtain unique and novel images in each epoch thus improving the model’s performance. Up sampling is done to ensure each label occurred at least 100 times in the dataset which increased the total diseased tongue images to 2729. Rotation, flipping and altering brightness saturation and hue is used for real time augmentation.

3.3. Deep Learning Models

Our Pipeline combines two different types of image classification methods, a binary classification for normal/diseased tongues and a disease label classifier for multilabel annotated images. We have used the AUCMEDI [49] platform to develop our pipeline in both cases we are using pretrained models to reduce time and cost of training a fresh model. Transfer learning is applied with frozen layers except the classification head. After 10 epochs of training the freezing is undone, so that the weights can be adapted to the new task.

The two architectures used are chosen to be compatible to the low resource requirement and possible to be deployed on android platform.

Diseased Tongue Detector

MobileNetV2 is used for binary classification to categorize between normal and unhealthy tongue images. We used 5-fold cross validation with bagging method for ensemble with Random Forest and Soft Majority Voting. Dataset with two classes is split into three parts as train, test model and test ensemble set. Ensemble dataset is used for bagging method by Random Forest classifier. The train set is further divided into train and validation set for 5-fold cross validation wherein, training-set is split into 5 parts and training is performed 5 times. Each time one of the five parts, a different one, is used for validation and the other four parts are used for training. Due to imbalance in images in the two categories class weights is computed and categorical focal loss is considered as the loss function. Testing of model performance done on the test data and evaluation metrics for training each fold are computed. Ensemble by 5-fold bagging is done by Random Forest and Soft Majority vote methods. Each model is trained for 50 epochs and dynamic learning rate set to maximum decrease to 1e-7 with factor of 0.1 in case of no improvement is monitored for validation loss, with patience of 5 epochs. Early stopping and Model checkpoint method is also used with patience level set as 12 epochs with no improvements in validation loss. Finally, all the individual folds F1 Scores along with ensemble models is compared to indicate the best performance of the models.

Disease Label Classifier

This is a multilabel classifier using Densenet121 architecture for the eight disease classes. Multilabel focal loss is used with categorical accuracy. Model is trained for 100 epochs with dynamic learning rate set to maximum decrease to 1e-7 with factor of 0.1 in case of no improvement is monitored for validation loss, with patience of 5 epochs. Early stopping and Model checkpoint method is also used stopping after 12 epochs with no improvements monitored.

The experiments were performed with hp Pavilion laptop with a 1.60 GHz intel i5 8^th generation processor and 8 GB of Ram. Training of the model was done on Google Colab Python 3 Google Compute Engine backend GPU.

3.4. Evaluation Metrices

Metrices considered for quantitative evaluation of the model on test data are Precision, Recall and F1-Score. Precision is number of true positives divided by the number of total positive predictions

$$Precision=\frac{True Positive \left(TP\right)}{True Positive\left(TP\right)+False Positive\left(FP\right)}$$

Recall measures the model’s ability to predict the positives, it is true positive divided by the true positive and false negative.

$$Recall=\frac{True Positive\left(TP\right)}{True Positive \left(TP\right)+False Negative\left(FN\right)}$$

F1 score is the harmonic mean of Precision and recall given by

$$F1Score=2\times \frac{Precision\times Recall}{Precision+Recall}$$

Categorical Accuracy calculates the percentage of predicted values that match the actual truth values. Calculation for the same is done as follows:

• First identify the index at which the maximum value occurs using argmax ().
• If it is the same for both predicted and true value, it is considered accurate.

Here since maximum value index is observed predicted value can be logit or probability function.

$$Categorical accuracy=\frac{accurately predicted records}{total number of records}$$

Reciever Operating Characteristics Curve

Receiver Operating characteristic (ROC) curve based on the calculation of average ROC for the 5-folds. Practically for each fold different False- and True-Positive-Rates exists and mean cannot be calculated hence, first aggregation of all False-Positive-Rates (FPR) into one vector is done serving as the x axis of average ROC curve. The True-Positive-Rate (TPR) are to be interpolated. The TPR is the corresponding y-points to the x-points that are already collected.

4. Results and Discussions

Disease risk detection model is trained on the dataset with healthy and unhealthy tongue images. Up sampling of the unhealthy image data is done to ensure at least 150 images in each of the 8 categories. Real time Image-augmentation used to increase the image-set artificially by adding small transformations to the original images such as rotations or changes of the contrast or saturation. With online image-augmentation the transformations are applied to each image when loaded with the data generator and need not be saved to disk. For multilabel disease classification evaluation one hot encoded file serves as interface defining true labels for the model.

4.1. Performance Analysis of Disease Risk Detector

The sequential training of the disease risk detector with 5-fold cross validation with MobileNet V2 architecture approximately took 4.5 hrs with 45 epochs for each fold on an average, when trained on shared GPU from COLAB. Disease classification model with DenseNet121 architecture took over 18 hrs to train. Performance results of the Disease Risk model w.r.t metrices considered on test model dataset of 485 images in all consisting of 361 unhealthy tongue images and 124 normal healthy tongue images are as shown in Table 3. Concise ROC Curve Figure 2, for the 5-folds along with ensemble techniques with mean value indicate appreciable performance of the model for binary classification.

Best model is saved for each fold of cross-validation. Predictions of each fold on a test ensemble dataset with 485 images are combined to form an ensemble. Implementation of the Random Forest ensemble done by picking random samples and features to build many decision trees on bootstrapped dataset by a repeated process. Prediction of new data is accomplished by bagging operation wherein final decision is the most common decision amongst all decision trees.

Aggregation done by Soft Majority Voting ensemble to create a new predictions matrix from the generated predictions. For each sample the category that has the max. value for this (ensemble) predictions-matrix is taken as the final prediction. Ensemble of the 5-fold predictions in this particular case with two classes does not show enhanced performance, F1 Scores for Random Forest and Soft Majority Voting are comparable to the individual folds F1 score as depicted in Figure 3.

4.1. Performance Analysis of Multi-Label Disease Classification Model

DenseNet 121 model with four dense blocks with [6,12,16,24] number of layers in each block is utilized for classification of diseases. Dataset consisted of 7 specific disease labels and one with others label under which all diseases which had less than 10 subjects falling in that particular category are placed. Model training stopped at 41 epochs. The performance parameters for the eight labels considered on a test data of 474 images is as in Table 4. ROC Curve in Figure 4 gives an idea of true positive predictions for each category.

It is observed that an average accuracy over 90% is achieved for each of the diseased class label. Some Sample results presented in Table 5 show the success of the DenseNet 121 model for disease classification. Highlighted rows in the table indicate inaccurate classification for 2 sample images. Class ‘others’ showed slightly lesser accurate results as also evident from ROC Curve and Performance indices.

Proposed model for Multi Disease Classification shows satisfactory results. The complete pipeline for disease risk detection and classification of multiple diseases can further be improved upon by adding more classification labels and using multiple models’ ensemble to achieve highest accuracy for all class labels.

Limitation of this work is small dataset with less variety of disease classes. Since only a single dense model is considered its performance accuracy for all class labels could not be uniform, this can be enhanced by incorporating ensemble learning method.

5. Conclusion and Future Work

Main challenging aspect of automating tongue analysis for disease diagnosis is quantifying the diagnostic features from the tongue image at par with the expert practitioner’s observations. In this study we introduced a powerful multi disease detection pipeline for tongue image analysis, which exploits ensemble learning to combine the predictions of 5 individually trained models for better performance. It utilizes strategies such as transfer learning, class weighting as imbalance in diff classes, extensive real time data augmentation, focal loss utilization and used two different deep learning models to predict the diseased conditions with multiple class labels. The most significant accomplishment from this study is that we could ascertain that mobile phone images can be successfully used for tongue image analysis, a step towards reducing the cost and expertise of a high-end sophisticated image capturing device. This opens opportunity to explore more enhanced models with a larger dataset for performance improvement.In our future work we aim to build more robust ensemble model capable to classify all possible disease label classification feasible using tongue analysis, this inherently includes within it need to compile an exhaustive tongue image dataset for research.