Understanding the ways knowledge concerning an external variable, or the reciprocal information of its parts, is distributed across the parts of a multivariate system can assist characterize and infer the underlying mechanics and function of the system. This goal has driven the development of several techniques for dissecting the elements of a set of variables’ combined entropy or for dissecting the contributions of a set of variables to the mutual information about the variable of interest. In actuality, this association and its modifications exist for any input signal and the widest range of Gaussian pathways, comprising discrete-time and continuous-time pathways in scalar or vector forms.

In a more general way, mutual information and mean-square error are the fundamental concepts of information theory and estimating theory, respectively. In contrast to the MMSE, which determines how precisely each input sample can be restored using the channel’s outcomes, the input-output mutual information is an estimation of whether the information can be consistently delivered over a channel given a specific input signal. An inactive functioning characterization for mutual information is provided by the substantial relevance of mutual information to estimate and filtering. Therefore, the significance of identity is not only obvious, but the link is also fascinating and merits an in-depth explanation [1,2,3]. Relations between the MMSE of the approximation of the output given the input and the localized actions of the mutual information at diminishing SNR are presented in [4]. [6] gives the idea about the probabilistic ratios of geometric characteristics of signal detection in Gaussian noise. Furthermore, whether in a continuous-time [5,6,7] or discrete-time setting [8] context, the likelihood ratio is difficult in the relationship between observation and estimation [9].

Considering the specific instance of parametric computation (or Gaussian inputs), correlations relating to causal and non-causal estimation errors have been investigated in [10,11], involving the limit on the loss owing to the causality restriction is specified. Knowing how data pertaining of an external parameter, or inversely related data within its parts, distributes across the parts of a multivariate system can assist categorize and determining the fundamental mechanics and functionality of the structure. This goal served as the impetus for the development of various techniques for decomposing the various elements of a set of parameters’ joint entropy [12,13] or to deconvolute the additions of a set of elements to the mutual information about a target variable [14]. These techniques can be used to examine a variety of intricate systems, including those in the physical distinctions domain, such as gene networks [15] or brain coding [16], as well as those in the social domain, such as selection agents [17] and community behavior [18]. They can also be used to analyze artificial agents [19]. Additionally, some new proposals diverge more significantly from the original framework, either through the adoption of novel principles, the consideration of the presence of detrimental elements linked to erroneous, or the implementation of joint entropy subdivisions in place of mutual information [20,21].

In the multivariate scenario, the challenges of breaking down mutual information into redundancy and complimentary sections have nevertheless been significantly increased. The novel redundancy determines that were initially developed are only defined for the bivariate situation [24,25], or allow negative components [26], whereas measurements of coordination are more readily extended to the multivariate case, especially when using the maximum entropy architecture [22,23]. By either utilizing the associations between lattices formed by various numbers of parameters or utilizing the multiple interactions between redundant lattices and information loss lattices, for which collaborative efforts are more actually defined, the study in [27] established two analogous techniques for constructing multivariate redundant metrics. The maximum entropy framework allows for a more straightforward generalization of the efficiency measurements to the multivariate case [24,25].

In the present study, we propose an extension of the bivariate Gaussian distribution technique to calculate multivariate redundant metrics inside the maximum entropy context. The importance of the maximum entropy approach in the bivariate scenario, where it offers constraints for the actual redundancy, unique information, and efficiency terms under logical presumptions shared by additional criteria, acts as the motivation for this particular focus [24]. The maximum entropy measurements, specifically, offer a lower limit for the actual cooperation and redundant terms and a higher limit for the actual specific information if it is presumed that a bivariate non-negative disintegration exists and that redundancy can be calculated from the bivariate distributions of the desired outcome with every source. Furthermore, if these bivariate distributions are consistent with possibly having little interaction under the previous hypotheses, then the maximum entropy decomposition returns not only boundaries but also the precise actual terms. Here, we demonstrate that, under similar presumptions, the maximum entropy reduction also plays this dominant role in the multivariate situation.

The remainder of this paper is organized as follows. A brief review of the geometry of the Gaussian distribution is reviewed in Section 2. The consecutive three sections deal with various important topics on information entropy with illustrative examples with emphasis on visualization of the information and discussion. In Section 3, continuous entropy/differential entropy is presented. In Section 4, the relative entropy (Kullback-Leibler divergence) is presented. Mutual information is presented in Section 5. Conclusions are given in Section 6.

In this section, the background relations on Gaussian distribution for different parametric point of view has been discussed. The exploratory analysis’s fundamental objective is to identify “the framework” in multivariate datasets. Ordinary least-squares regression and principal component analysis (PCA), respectively, offer typical measurements for dependency (the predicted connection between particular components) and rigidity (the degree of prominence of the probability density function (pdf) around a low-dimensional axis) for bivariate Gaussian distributions. Mutual information, an established measure of dependency, is not an accurate indicator of rigidity since it is not invariant with an opposite rotation of the parameters. For bivariate Gaussian distributions, a suitable rotating invariant compactness measure is constructed and demonstrated to reduce the corresponding PCA measure.

The Gaussian pdf (a) does not have a framework in either of the above-described definitions and represents the independent variables without any settling around a lower-dimensional region. The Gaussian pdf (b), on the other hand, has greater variance along one axis over another. Despite being independent, their combined pdf is small. Although the variables are associated and therefore likewise characterized by dependency, the Gaussian pdf (c) is equally focused around one dimension as is (b).

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula and rather just assumed it was the correct continuous analog of discrete entropy, but it is not.[1]: [181–218]. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP and one that loses its fundamental association with discrete entropy.

In the following discussion, differential entropy, and relative entropy are measured in bits, which is used in the definition. Instead, if ln is used, it is then measured in nats, and the only difference in the expression is the ${\log }_{2}e$factor.

In this section, various important issues regarding the relative entropy (Kullback-Leibler divergence will be delivered. Despite the aforementioned flaws, there is a possibility of information theory in the continuous case. A key result is that definitions for relative entropy and mutual information follow naturally from the discrete case and retain their usefulness.

The relative entropy is a type of statistical distance that provides a measure of how one probability distribution $f_X$ is different from a second, reference probability distribution $g_X$, denoted as

Detailed derivation is provided in Appendix E. The relative entropy between two Gaussian distributions with different means and variances are given by

Notice that the relative entropy here is measured in bits where ${\log }_2$ is used in the definition. In stead, if ln is used, it would be measured in nats. The only difference in the expression is the ${\log }_{2}e$ factor. Several conditions are discussed.

(1) If ${\sigma }_1={\sigma }_2=\sigma$, $D_{KL}(f||g)=\frac{1}{2}{\left(\frac{{\mu }_1-{\mu }_2}{\sigma }\right)}^2{\log }_{2}e$, which is 0 when${\mu }_1={\mu }_2$.

(2) If ${\sigma }_1={\sigma }_2=1$, $D_{KL}(f||g)=\frac{1}{2}{({\mu }_1-{\mu }_2)}^2\cdot {\log }_{2}e$, which is a even function with a minimum value of 0 when${\mu }_1={\mu }_2$.

- If ${\mu }_2=0$, $D_{KL}(f||g)=\frac{1}{2}{\mu }_1^2{\log }_{2}e$, it is a function of ${\mu }_1$ concave upward.

- If ${\mu }_1=0$, $D_{KL}(f||g)=\frac{1}{2}{\mu }_2^2{\log }_{2}e$, it is a function of ${\mu }_2$ concave upward.

(3) If ${\mu }_1={\mu }_2$, $D_{KL}(f||g)=\frac{1}{2}\left[\ln \left(\frac{{\sigma }_2^2}{{\sigma }_1^2}\right)+\frac{{\sigma }_1^2}{{\sigma }_2^2}-1\right]\cdot {\log }_{2}e$

- When ${\sigma }_2=1$, $D_{KL}(f||g)=\frac{1}{2}\left[\ln \left(\frac{1}{{\sigma }_1^2}\right)+{\sigma }_1^2-1\right]\cdot {\log }_{2}e$

- When ${\sigma }_1=1$, $D_{KL}(f||g)=\frac{1}{2}\left[\ln \left({\sigma }_2^2\right)+\frac{1}{{\sigma }_2^2}-1\right]\cdot {\log }_{2}e$

Sensitivity analysis of the relative entropy due to change of variances and means. The gradient of $D_{KL}(f||g)$ given by can be calculated where the calculation deals with partial derivatives where the cahin rule is involved. Based on the relation$\frac{d}{dx}\ln x=\frac{1}{x}$, we have and the following derivatives are obtained.

(1) $\frac{\partial D_{KL}}{\partial {\sigma }_1}=\frac{\partial }{\partial {\sigma }_1}\left[\ln \left(\frac{{\sigma }_2^2}{{\sigma }_1^2}\right)+\frac{{\sigma }_1^2}{{\sigma }_2^2}\right]\cdot \frac{1}{2}{\log }_{2}e=\left(\frac{{\sigma }_1}{{\sigma }_2^2}-\frac{1}{{\sigma }_1}\right)\cdot {\log }_{2}e$(2) $\frac{\partial D_{KL}}{\partial {\sigma }_2}=\frac{\partial }{\partial {\sigma }_2}\left[\ln \left(\frac{{\sigma }_2^2}{{\sigma }_1^2}\right)+\frac{{\sigma }_1^2}{{\sigma }_2^2}+\frac{{({\mu }_1-{\mu }_2)}^2}{{\sigma }_2^2}\right]\cdot \frac{1}{2}{\log }_{2}e=\left[\frac{1}{{\sigma }_2}-\frac{{\sigma }_1^2}{{\sigma }_2^3}-\frac{{({\mu }_1-{\mu }_2)}^2}{{\sigma }_2^3}\right]\cdot {\log }_{2}e$(3) $\frac{\partial D_{KL}}{\partial {\mu }_1}=\frac{\partial }{\partial {\mu }_1}{\left(\frac{{\mu }_1-{\mu }_2}{{\sigma }_2}\right)}^2\cdot \frac{1}{2}{\log }_{2}e=\left(\frac{{\mu }_1-{\mu }_2}{{\sigma }_2^2}\right)\cdot {\log }_{2}e$(4) $\frac{\partial D_{KL}}{\partial {\mu }_2}=\frac{\partial }{\partial {\mu }_2}{\left(\frac{{\mu }_1-{\mu }_2}{{\sigma }_2}\right)}^2\cdot \frac{1}{2}{\log }_{2}e=\left(\frac{{\mu }_2-{\mu }_1}{{\sigma }_2^2}\right)\cdot {\log }_{2}e$For optimality for each of the above cases, we have

Mutual information is one of many quantities that measures how much one random variables tells us about another. It is a dimensionless quantity with (generally) units of bits, and can be thought of as the reduction in uncertainty about one random variable given knowledge of another. The mutual information $I(X;Y)$between two variables with joint pdf $f_{XY}(x,y)$is given by

The mutual information between the random variables X and Y has the following relation where and implying that $h(X)\ge h(X|Y)$ and $h(Y)\ge h(Y|X)$. The mutual information of a random variable with itself is the self information, which is the entropy. High mutual information indicates a large reduction in uncertainty; low mutual information indicates a small reduction; and zero mutual information between two random variables, $I(X;Y)=0$, meaning that the variables are independent. In such case, $h(X)=h(X|Y)$and $h(Y)=h(Y|X)$.