2.1. Principal Components and the Projection Matrix

• Data normalization: is the first step of the calculation and we subtract the mean of each feature vector from the corresponding vector entries. This is done to ensure that the data is centered around zero. It means that, the mean of every feature vector is 0. The equation for data normalization with the help of the feature vector mean is:

  $$x_{ij}\leftarrow x_{ij}-{\mu }_j,$$

  (1)

  where $x_{ij}$ is the $i-th$ feature value, where $i=1,....,n$, of the $j-th$ feature (variable) vector, for $j=1,....,k$, and ${\mu }_j$ is the mean of this feature vector.
• Covariance matrix: measures the statistical relationship between the features in the entire dataset [15]. For this purpose we develop the equation below and calculate the $F_{pj}$ entry of the features’ covariance matrix of the training data matrix ${\mathbf{D}}_{\mathbf{t}}$:

  $$F_{pj}=\frac{1}{n-1}\sum _{i=1}^n(x_{ij}-{\mu }_j){(x_{ip}-{\mu }_p)}^T.$$

  (2)
  It follows from Eq. 2, the covariance matrix of the ${\mathbf{D}}_{\mathbf{t}}$ features is of the form:

  $$\mathbf{F}=\left\{F_{pj}\right\},p=1,...,k,j=1,...,k.$$

  (3)
  Now we calculate the Eigenvectors ${\mathbf{u}}_{\mathbf{j}}$ and the Eigenvalues ${\lambda }_i$ of the covariance matrix $\mathbf{F}$ of the database features as follows:

  $$\mathbf{F}{\mathbf{u}}_{\mathbf{j}}={\lambda }_j{\mathbf{u}}_{\mathbf{j}},|\mathbf{F}-\lambda \mathbf{I}|=0,(\mathbf{F}-\lambda \mathbf{I}){\mathbf{u}}_{\mathbf{j}}=\mathbf{0},$$

  (4)

  where we denote the $k\times k$ identity with $\mathbf{I}$.
• Projection: At the present step we construct the matrix $\mathbf{U}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{r}}$ whose columns are the normalized Eigenvectors of the covariance matrix $\mathbf{F}$ of the dataset ${\mathbf{D}}_{\mathbf{t}}$ features:

  $$\mathbf{U}=[{\mathbf{u}}_{\mathbf{1}},{\mathbf{u}}_{\mathbf{2}},...,{\mathbf{u}}_{\mathbf{r}}],$$

  (5)

  where every Eigenvector ${\mathbf{u}}_{\mathbf{j}}\in {\mathbf{R}}^{\mathbf{k}}$, and r is the rank of ${\mathbf{D}}_{\mathbf{t}}$. Also, let us arrange the vector columns ${\mathbf{u}}_{\mathbf{j}}$ from left to right according to the decreasing Eigenvalue of the corresponding Eigenvector. We consider the largest one of them at the left most position, if we think of the Eigenvalues as a decreasing sequence of consecutive numbers. The matrix $\mathbf{U}$ is applied to project the training dataset ${\mathbf{D}}_{\mathbf{t}}$ onto the new subspace spanned by the Eigenvectors $u_i,i=1,...,r$. The equation that conducts the projection is [15]:

  $${\mathbf{y}}_{\mathbf{i}}={\left({\mathbf{U}}^T{\mathbf{x}}_{\mathbf{i}}\right)}^T={\mathbf{x}}_{\mathbf{i}}^T\mathbf{U},$$

  (6)

  where ${\mathbf{y}}_{\mathbf{i}}\in {\mathbf{R}}^{\mathbf{r}}$ is the new low-dimensional representation of the i-th training sample ${\mathbf{x}}_{\mathbf{i}}\in {\mathbf{R}}^{\mathbf{k}}$, where ${\mathbf{x}}_{\mathbf{i}}$ is a vector row in ${\mathbf{D}}_{\mathbf{t}}$ whose rank $r\le k$.
  We consider that an Eigenvector is stronger than another one if the former has a bigger Eigenvalue than the latter. Hence, we select from ${\mathbf{U}}^{\mathbf{T}}$ the $m<r$ strongest components (Eigenvectors) ${\mathbf{u}}_{\mathbf{j}}$, which appear as the top m rows of the matrix. Using the strongest components, as vector columns, we construct the matrix $\mathbf{T}\in {\mathbf{R}}^{\mathbf{k}\times \mathbf{m}}$ which transforms (maps) the original dataset ${\mathbf{D}}_{\mathbf{t}}$ into ${\mathcal{D}}_{\mathbf{t}}\in {\mathbf{R}}^{\mathbf{n}\times \mathbf{m}}$, which we call the transformed data:

  $${\mathbf{D}}_{\mathbf{t}}\mathbf{T}={\mathcal{D}}_{\mathbf{t}}.$$

  (7)

2.2. Finding the Distilled Vectors

2.3. Minimum Number Training Samples (MNTS)

• Assume we are looking for an accuracy $a\in (0,1]$. Follows that $\alpha =1-a$ and the confidence level $C=1-\alpha /2$ ;
• Using the confidence level C we determine $Z_{\alpha /2}$ from the standard normal distribution table [17];
• Select the success and failure rate and denote them respectively with p and q and calculate m from

  $$\frac{\alpha }{\sqrt{\frac{pq}{m}}}\ge Z_{\frac{\alpha }{2}}$$

  (9)
• Consider a training set with two classes (the number of benign $\left|B\right|$ and the number of malignant $\left|M\right|$) $\left|B\right|+\left|M\right|=\left|S\right|$ total number of samples.
• Calculate $\frac{\left|M\right|}{\left|S\right|}=c_1$, $\frac{\left|B\right|}{\left|S\right|}=c_2$
• $m_M$ and $m_B$ are respectively the number of malignant and the number of benign distilled vectors. $m_M=n·c_1$, $m_B=n·c_2$ such that $m_M+m_B=m$.

3.1. Logistic Regression (LR)

3.2. Support Vector Machine (SVM)

3.3. Neural Network (NN)

4.1. Original Datasets

• The first one we call Skin Lesion (SL) Dataset which comprises 162 observations (samples S) of 5D feature vectors ($B,C,D,A_B^M,A_C^M$) extracted by an active contour in [26] from skin lesion images with a ground truth [25]. The SL dataset contains 100 benign samples (skin lesion feature vectors labeled by B) and 62 malignant observations (labeled by M).
• The Diabetes (D) Dataset [27] consists of 768 feature vectors with a dimension 8 ($8D$). The vectors are distributed in two sets labeled by P if the corresponding vector indicates Diabetes. Otherwise, the vector is labeled by N, which means that the vector indicates NON-Diabetes. The class P which suggests Diabetes consists of 268 vectors, while the class N contains 500 vectors of dimension 8.
• The Heart Disease (HD) Dataset [28] consists of 1025 feature vectors of dimension 13 features. The data samples are distributed in two classes $0=Negative\left(N\right)$ and $1=Positive\left(P\right)$. The N samples in the HD datasets are 499 while the P are 526.
• Breast Cancer (BC) Dataset [29] contains total of 569 feature vectors (observations) of dimension 30D . Out of them 347 labeled as benign-B while the remaining 212 are determined, by the medical experts, to be malignant-M.

4.2. Augmentation of the Original Training Data

4.3. Model Training

4.4. Cross-Validation

4.5. Classification Metrics

• Accuracy : Accuracy measures the overall correctness of the model’s predictions by calculating the ratio of correctly predicted instances to the total number of instances.

  $$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$
• Sensitivity (True Positive Rate): Sensitivity, also known as recall or true positive rate, measures the model’s ability to correctly identify positive instances out of all the actual positive instances.

  $$\mathrm{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$
• Specificity (True Negative Rate): Specificity calculates the model’s ability to correctly identify negative instances out of all the actual negative instances.

  $$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$