The following notations will be used in this paper: Real scalar variables, whether mathematical or random, will be denoted by lower-case letters such as $x,y$. Real vector/matrix variables, whether mathematical or random, whether square matrices or rectangular matrices, will be denoted by capital letters such as $X,Y$. Scalar constants will be denoted by lower-case letters such as $a,b$ and vector/matrix constants will be denoted by $A,B$ etc. A tilde will be used to designate variables in the complex domain such as $\tilde{x},\tilde{y},\tilde{X},\tilde{Y}$. No tilde will be used on constants. When Greek letters and other symbols appear, the notations will be explained then and there. Let $X=\left(x_{ij}\right)$ be a $p\times q$ matrix where the elements are functionally independent or distinct real scalar variables. Then, the wedge product of differentials will be defined as $\mathrm{d}X={\wedge }_{i=1}^p{\wedge }_{j=1}^q\mathrm{d}{x}_{ij}$. When x and y are real scalars, then the wedge product of their differentials is defined as $\mathrm{d}x\wedge \mathrm{d}y=-\mathrm{d}y\wedge \mathrm{d}x$ so that $\mathrm{d}x\wedge \mathrm{d}x=0,\mathrm{d}y\wedge \mathrm{d}y=0$. For a square matrix A the determinant will be denoted as $\left|A\right|$ or as $\det \left(A\right)$. When A is in the complex domain, then the absolute value of the determinant or modulus of the determinant will be denoted as $\left|\det \right(A\left)\right|$. If $\left|A\right|=a+ib,i=\sqrt{(-1)},a,b$ real scalars, then $\left|\det \left(A\right)\right|=\sqrt{(a^2+b^2)}$. If $\tilde{X}$ is in the complex domain, then one can write $\tilde{X}=X_1+i{X}_2,i=\sqrt{(-1)},X_1,X_2$ real, then the wedge product of differentials in $\tilde{X}$ will be defined as $\mathrm{d}\tilde{X}=\mathrm{d}{X}_1\wedge \mathrm{d}{X}_2$. We will consider only real-valued scalar functions in this paper. ${\int }_{X}f\left(X\right)\mathrm{d}X$ will denote integral over X of the real-valued scalar function $f\left(X\right)$ of X. When $f\left(X\right)$ is a real-valued scalar function of X, whether X is scalar or vector or matrix in the real or complex domain, and if $f\left(X\right)\ge 0$ for all X and ${\int }_{X}f\left(X\right)\mathrm{d}X=1$, then $f\left(X\right)$ will be defined as a density or statistical density. When a square matrix X is positive definite then it will be denoted as $X>O$ where $X=X^{\prime }$, a prime denoting the transpose. Conjugate transpose of any matrix $\tilde{Y}$ in the complex domain will be written as ${\tilde{Y}}^{*}$. When a square matrix $\tilde{X}$ is in the complex domain and if $\tilde{X}={\tilde{X}}^{*}$ then $\tilde{X}$ is Hermitian. If $\tilde{X}={\tilde{X}}^{*}>O$ then $\tilde{X}$ is called Hermitian positive definite. When $Y>O$, then ${\int }_A^{B}f\left(X\right)\mathrm{d}X$ means the integral of the real-valued scalar function $f\left(X\right)$ over the real positive definite matrix X such that $X>O,A>O,B>O,X-A>O,B-X>O$ (all positive definite), where $A>O$ and $B>O$ are constant matrices, and similar notation and interpretation in the complex domain also. In order to avoid multiplicity of numbers, the following procedure will be used. For a function number or equation number in the complex domain, corresponding to the same in the real domain, a letter c will be affixed to the function number and section number of the equation number. For example $f_{1c}\left(\tilde{X}\right)$ will correspond to $f_1\left(X\right)$ in the real domain and equation number $(2c.5)$ will correspond to $\left(2.5\right)$ in the real domain. This notation will enable a reader to recognize a function or equation in the complex domain instantly by recognizing the subscript c. Other notations will be explained whenever they occur for the first time.

Matrix-variate statistical distributions are widely used in all types of disciplines such as Statistics, Physics, Communication Theory, Engineering problems. A matrix-variate density where a trace with an exponent enters into the density as a product and when the exponential trace has an arbitrary power, known as Kotz’ model in the literature, is widely used in the analysis of data coming from various areas such as multi-look return signals in radar and sonar, see for example, [2] regarding the analysis of PolSAR (Polarimetric Synthetic Aperture Radar) data. Kotz’ model is a generalization of the basic matrix-variate Gaussian model or it can also be considered as a generalization of matrix-variate gamma model or Wishart model. When analysing radar data, it is found that Gaussian-based models fit well when the surface is disturbance-free. It is found that Gaussian-based models are not appropriate in certain regions such as urban area, sea surface, forests etc, see for example, [3–6]. Hence, we will consider some non-Gaussian or non-Wishart models also in this paper, along with Gaussian-based models. In most of the applications in engineering areas, each scalar variable has two components, such as time and phase, so that a complex variable is very appropriate to represent such a scalar variable. Hence, it is found that distributions in the complex domain are more important in applications in physical sciences and engineering areas. When a statistical density is used in any applied problem, computation of the normalizing constant there is the most important step because when studying all sorts of properties of such a model, the computations naturally follow the format of the evaluation of the normalizing constant in that model. Explicit evaluation of the normalizing constant in a general model, often referred to also as a Kotz’ model in the real domain, does not seem to be available in the literature. Normalizing constant in the general model in the real domain appearing in [7], which the authors claim to have been available elsewhere in earlier literature, seems to be the one widely used in all the applications where Kotz’ model in the real domain is used. But, unfortunately the normalizing constant quoted in [7] does not seem to be correct. Kotz’ type model in the complex domain does not seem to be available in the literature and the normalizing constant therein does not seem to be available also. Hence, one of the aims of this paper is to give the derivation of the normalizing constant in the general model in detail, in the real and complex domains, and also to extend the ideas to Mathai’s pathway family [1], namely matrix-variate gamma, type-1 beta and type-2 beta families of densities. Since the derivation of the normalizing constant is the most important step in the construction of any statistical model, various matrix-variate models are listed in this paper by showing the computations of the normalizing constants in each case. Some applications Kotz’ model in the real domain may be seen from [8–11].

This paper is organized as follows: Section 1 contains the introductory material. Section 2 gives explicit evaluation of the normalizing constant in an extended matrix-variate gamma type or Gaussian type or Wishart type or Kotz type model, both in the real and complex domains, and then deals with multivariate and matrix-variate extended Gaussian and gamma type distributions. Section 3 examines extended matrix-variate type-2 beta type models in the real and complex domains. Section 4 contains extended matrix-variate models of the type-1 beta type in the real and complex domains. Throughout the paper, the results in the complex domain are listed side by side with the corresponding results in the real domain. Detailed derivations are done for the real domain cases only, since most of the steps in the complex domain are parallel to those in the real domain.

Let us start with an example of the evaluation of an integral in the real domain which will show the different types of hurdles to overcome to achieve the final result. Let $X=\left(x_{ij}\right)$ be a $p\times q,p\le q$ matrix of rank p where the $pq$ elements $x_{ij}$’s are functionally independent (distinct) real scalar variables. Suppose that we wish to evaluate the following integral, where $f\left(X\right)$ is a real-valued scalar function of the $p\times q$ matrix X, integral over X and the wedge product of differentials $\mathrm{d}X$ are already explained in Section 1: where $\delta >0,M=E\left[X\right],\Re \left(\eta \right)>0,\Re \left(\gamma \right)>-\frac{q}{2}+\frac{p-1}{2},A>O$ is $p\times p$ and $B>O$ is $q\times q$ positive definite constant matrices, $A^{\frac{1}{2}}$ is the positive definite square root of the positive definite matrix $A>O$, $E[·]$ denotes the expected value of $[·]$ and $\Re (·)$ means the real part of $(·)$. The first step here is to simplify the matrix $A^{\frac{1}{2}}(X-M)B{(X-M)}^{\prime }{A}^{\frac{1}{2}}$ into a convenient form by making a transformation $Y=A^{\frac{1}{2}}(X-M)B^{\frac{1}{2}}\Rightarrow \mathrm{d}Y={\left|A\right|}^{\frac{q}{2}}{\left|B\right|}^{\frac{p}{2}}\mathrm{d}X$ from the Lemma 2.1 given below by observing that $\mathrm{d}(X-M)=\mathrm{d}X$ since M is a constant matrix. The corresponding integral in the complex domain is the following: where $A=A^{*}>O,B=B^{*}>O$ (both Hermitian positive definite), A is $p\times p$, B is $q\times q$ and $\tilde{M}=\tilde{X}$. The transformation in the complex case is $\tilde{Y}=A^{\frac{1}{2}}(\tilde{X}-\tilde{M})B^{\frac{1}{2}}\Rightarrow \mathrm{d}\tilde{Y}=|\det \left(A\right){|}^q{\left|\det \left(B\right)\right|}^p\mathrm{d}\tilde{X}$.

The proof of Lemma 2.1 and other lemmas to follow, may be seen from [12]. When a $p\times p$ matrix X is symmetric, $X=X^{\prime }$ then we have a companion result to Lemma 2.1 which will be stated next.

Now, under Lemma 2.1, $\left(2.1\right)$ reduces to an integral over Y. Let us denote it as $f_1\left(Y\right)$. Then, The corresponding integral in the complex case is the following:

The function $f_1\left(Y\right)$ in the real domain when $\gamma =0,\eta \ne 0,\delta \ne 1$ is often referred to as Kotz’ model by most of the authors who use such a model. When the exponent of the determinant $\gamma \ne 0$ then the evaluation of the integral over $f_1\left(Y\right)$ is very difficult, which will be seen from the computations to follow. When $\gamma \ne 0$, and in the real domain, [7] calls the model also as Kotz’ model but the normalizing constant given by them and claimed to be available in earlier literature, does not seem to be correct. The correct normalizing constant and its evaluation in the real and complex domains will be given in detail below. Since $f_2\left(Y\right)$ involves a determinant and a trace where the determinant is a product of eigenvalues and trace is a sum, two elementary symmetric functions, if there a transformation involving elementary symmetric functions then one can handle the determinant and trace together. This author does not know any such transformation. Going through eigenvalues does not seem to be a good option because the Jacobian element will involve a Vandermonde determinant and not very convenient to handle. The next possibility is triangularization and in this case also one can make the determinant as a product of scalar variables and trace as a sum. Then, one can use a general polar coordinate transformation so that the trace becomes a single variable, namely the radial variable r and in the product r and sine and cosine product will be separated also. Hence, this approach will be a convenient one. Continuing with the evaluation of $\left(2.2\right)$ in the real case, we have the following situations: if $\delta =1$ and $\eta =0$, or $\gamma =0$, then one would immediately convert $\mathrm{d}X$ into $\mathrm{d}S,S=X{X}^{\prime }$ and integrate out by using a real matrix-variate gamma integral in the case of $\eta =0$ and $\delta =1$ and integrate out by using the scalar variable gamma integral in the case $\gamma =0$. This conversion can be done with the help of Lemma 2.3 given below.

But in our $\left(2.2\right)$, both the determinant and trace enter as multiplicative factors and there is an exponent $\delta >0$ for the exponential trace. In order to tackle this situation, we will convert $\mathrm{d}X$ to $\mathrm{d}T$, where T is a lower triangular matrix, by using Theorem 2.14 of [12] which is restated here as a lemma. The idea is that in this case $|X{X}^{\prime }|=|T{T}^{\prime }|$ becomes product of the squares of the diagonal elements in T only and $\mathrm{tr}\left(T{T}^{\prime }\right)$ is a sum of squares also. This conversion can also be achieved by converting $\mathrm{d}S$ of Lemma 2.3 to a $\mathrm{d}T$ by using another result, where T is lower triangular.

Let us consider the evaluation of $\left(2.2\right)$ in the real case first. Converting $\mathrm{d}Y$ in $\left(2.2\right)$ to $\mathrm{d}T$ by using Lemma 2.4, the integral part of $\left(2.2\right)$ over Y, is the following, denoted by $f_2\left(T\right)$: The corresponding equation in the complex domain is the following: Note that, in the real case where in ${\sum }_{j=1}^p{t}_{jj}^2$ there are p terms and second sum has $p(p-1)/2$ terms, thus a total of $k=p(p+1)/2$ terms. The corresponding quantity in the complex domain is the following: where ${\tilde{t}}_{jk}=t_{jk1}+i{t}_{jk2},i=\sqrt{(-1)},t_{jk1},t_{jk2}$ real, and in the first sum there are p square terms but in the sum ${\sum }_{j>k}{\left|{\tilde{t}}_{jk}\right|}^2$ there are a total of $2\left[\frac{p(p-1)}{2}\right]=p(p-1)$ square terms, thus a total of $p^2$ square terms in the complex case.

Let us consider a polar coordinate transformation in the real case on all the $k=p(p+1)/2$ terms by using the transformation on page 44 of [12] which is restated here for convenience, that is $\{t_{11},t_{22},...,t_{pp},t_{21},...,t_{pp-1}\}\to \{r,{\theta }_1,...,{\theta }_{k-1}\}$, $k=p(p+1)/2$. for $-\frac{\pi }{2}<{\theta }_j\le \frac{\pi }{2},j=1,...,k-2;-\pi <{\theta }_{k-1}\le \pi$ for $k=p(p+1)/2$ in the real case and $k=p^2$ in the complex case. The structure of the polar coordinate transformation in the complex case remains as in the real case, we will denote it as (2c.4), the only change is that in the real case $k=p(p+1)/2$ and in the complex case $k=p^2$. The Jacobian of the transformation in the real case is and in the complex case the Jacobian is given by for the same ranges for ${\theta }_j$’s as in the real case, but in the complex case $k=p^2$.

The normalizing constant c in the real case coming from (2.3) is quoted in [7] by citing earlier works. But none of them seems to have given the evaluation of the integral in (2.3) explicitly. The normalizing constant c given in [7] does not seem to be correct. Since the integral in (2.3) appears in very many places as Kotz’ integral, and used in many disciplines, a detailed evaluation of the integral in (2.3) is warranted. Also, none seems to have given $\tilde{c}$ in the complex case. Hence, the evaluations of c and $\tilde{c}$ in the real and complex cases will be given here in detail.

Note that ${\sum }_{ij}{t}_{ij}^2=r^2$. From the Jacobian part, the factor containing r is $r^{k-1}={\left(r^2\right)}^{\frac{k}{2}-\frac{1}{2}}$. In the product ${\prod }_{j=1}^p{t}_{jj}^2$ each $t_{jj}^2$ contains a $r^2$. Also, the Jacobian part ${\prod }_{j=1}^p|t_{jj}{|}^{q-j}={\prod }_{j=1}^p{\left(t_{jj}^2\right)}^{\frac{q}{2}-\frac{j}{2}}={\left|T{T}^{\prime }\right|}^{\frac{q}{2}-\frac{j}{2}}$. Collecting all r, the exponent of $r^2$ in the real case is the following: Then, integration over r gives the following: for $\Re \left(\gamma \right)>-\frac{q}{2},\Re \left(\eta \right)>0,\alpha >0,\delta >0$. The corresponding integral over r in the complex domain is the following: for $\Re \left(\gamma \right)>-q,\delta >0,\alpha >0,\Re \left(\eta \right)>0$.

Consider the integration of factors containing the ${\theta }_j$’s in the real case. These ${\theta }_j$’s come from ${\prod }_{j=1}^p{t}_{jj}^2$ and from the Jacobian part. Consider ${\theta }_1$. The exponent of ${sin}^2{\theta }_1$ is $\gamma +\frac{q}{2}-\frac{1}{2}$. The exponent of ${cos}^2{\theta }_1$ is $(\gamma +\frac{q}{2}-\frac{2}{2})+(\gamma +\frac{q}{2}-\frac{3}{2})+...+(\gamma +\frac{q}{2}-\frac{p}{2})=(p-1)(\gamma +\frac{q}{2})-\frac{p(p+1)}{4}+\frac{1}{2}$ and the part coming from the Jacobian $|cos{\theta }_1{|}^{k-1-1}$. Note that $|cos{\theta }_1{|}^{k-1-1}={\left({cos}^2{\theta }_1\right)}^{\frac{p(p+1)}{4}-\frac{3}{2}}|cos{\theta }_1|$. Then, the integral over over ${\theta }_1$, denoting the integral over ${\theta }_1$ as $I_{{\theta }_1}$, gives the following, where in all the integrations over the ${\theta }_j$’s to follow, we will use the transformations $x=sin{\theta }_j,u=x^2$: Now, collecting all factors containing ${\theta }_2$ and proceeding as in the case of ${\theta }_1$ we have the following result for the integral over ${\theta }_2$: Note that the denominator gamma in $\left(ii\right)$ cancels with one numerator gamma in $\left(i\right)$. The pattern of cancellation of the denominator gamma in the next step with one numerator gamma in its previous step will continue leaving only one factor in the numerator and no factor in the denominator, except the very first step involving $\left(i\right)$ and $\left(ii\right)$ where the first denominator gamma, namely $\Gamma \left(p(\gamma +\frac{q}{2})\right)$ is left out. When, integrating ${\theta }_{p-1}$, we have the following: Note that when considering ${\theta }_p$ there is no cosine factor coming from $t_{pp}$ and the cosine factor comes only from the Jacobian part. We can see that Again, the denominator gamma in $\left(iv\right)$ cancels with one numerator gamma in $\left(iii\right)$. This pattern will continue for $k=p+1,p+2,...$. For the integrals over ${\theta }_j,j=p+1,p+2,...$ the only contribution is from the Jacobian part, no sine factor will be there. Consider ${\theta }_{p+1}$. We see that Again, cancellation will hold. Now, consider a few last cases of ${\theta }_j$. For $j=\frac{p(p+1)}{2}-3=k-3$ we have and for $j=k-2$ and the last ${\theta }_j$ goes from $-\pi$ to $\pi$ and no contribution from the Jacobian part and hence Note that starting from $j=p+1$ to $j=k-2$ the gamma factor left in the numerator is $\Gamma \left(\frac{1}{2}\right)=\sqrt{\pi }$. There are $\frac{p(p-1}{2}-2$ such factors and the last one is $2\pi$, thus the product is $2{\pi }^{\frac{p(p-1)}{4}}$. For $j=1,...,p$ the factors left out in the numerator are $\Gamma (\gamma +\frac{q}{2})\Gamma (\gamma +\frac{q}{2}-\frac{1}{2})...\Gamma (\gamma +\frac{q}{2}-\frac{p-1}{2})$ and for $j=p+1,...,k-1$ we have ${\pi }^{\frac{p(p-1)}{4}}$ giving ${\Gamma }_p(\gamma +\frac{q}{2}),\Re \left(\gamma \right)>-\frac{q}{2}+\frac{p-1}{2}$. For $j=1$ there is one gamma left in the denominator, namely, $\Gamma \left(p(\gamma +\frac{q}{2})\right)$. Taking the product of integral over all ${\theta }_j$’s in the real case is where ${\Gamma }_p(·)$ is the real matrix-variate gamma defined in Lemma 2.3.

The sine and cosine functions come from the transformations corresponding to $t_{11},...,t_{pp}$, from the Jacobian when going from $\tilde{X}$ to $\tilde{T}$ and from the Jacobian in the polar coordinate transformation. The Jacobian part in the polar coordinate transformation is Collecting factors containing ${\theta }_1$, observing that $sin{\theta }_1$ comes from $t_{11}$ and $cos{\theta }_1$ comes from $t_{22},...,t_{pp}$ and the Jacobian part. The exponent of ${sin}^2{\theta }_1$ is $\gamma +q-\frac{1}{2}$ and the exponent of ${cos}^2{\theta }_1$ is $(p-1)(\gamma +q+\frac{1}{2})-\frac{p(p+1)}{2}+1+\frac{p^2}{2}-\frac{1}{2}-1=(p-1)(\gamma +q)-1$. In all the integrals to follow, ${\int }_{-\frac{\pi }{2}}^{\frac{\pi }{2}}(·)\mathrm{d}\theta =2{\int }_0^{\frac{\pi }{2}}(·)\mathrm{d}\theta$ due to evenness of the integrand. Then, we will use the transformations $x=sin\theta ,u=x^2$, the steps parallel to those in the real case. Therefore for $\Re \left(\gamma \right)>-q$. Collecting the factors containing ${\theta }_2$, we note that the exponent of ${sin}^2{\theta }_2$ is $\gamma +q-2+\frac{1}{2}$ and the exponent of ${cos}^2{\theta }_2$ is $(p-2)(\gamma +q+\frac{1}{2})-\frac{p(p+1)}{2}+1+2+\frac{p^2}{2}-2=(p-2)(\gamma +q)$. Hence, Note that $\Gamma \left(\right(p-1\left)\right(\gamma +q\left)\right)$ from the denominator of $\left(iii\right)$ cancels with the same in the numerator of $\left(ii\right)$ leaving one gamma, namely $\Gamma (\gamma +q)$ in the numerator of $\left(ii\right)$ and one gamma, namely $\Gamma \left(p\right(\gamma +q\left)\right)$ in the denominator of $\left(ii\right)$. The pattern of cancellation of the gamma in the denominator of a step canceling with a gamma in the numerator of the previous step will continue as seen in the real case. Let us check for $j=p,j=p+1$ to see whether the pattern is continuing, where in $j=p+1$ there is no contribution of sine function, the only cosine function coming is from the Jacobian part. For $j=p$ we have For $j=p+1$ The pattern of cancelation is continuing. But, starting from $j=p+1,...,p^2-2$ the factor left out in the numerator is $\Gamma \left(\frac{1}{2}\right)=\sqrt{\pi }$ and the last factor gives $2\pi$ because the range here is $-\pi <{\theta }_{p^2-1}\le \pi$ and hence the factors left out in the numerator are ${\left(\sqrt{\pi }\right)}^{p(p-1)}\Gamma (\gamma +q)\Gamma (\gamma +q-1)...\Gamma (\gamma +q-p+1)={\tilde{\Gamma }}_p(\gamma +q),\Re \left(\gamma \right)>-q+p-1$ and one gamma, namely $\Gamma \left(p\right(\gamma +q\left)\right)$ is left out in the denominator of the integration over ${\theta }_1$. Hence, the integration over all sine and cosine functions in the complex case is where ${\tilde{\Gamma }}_p(·)$ is the complex matrix-variate gamma function defined in Lemma 2.3. Then the final result of integration over r and integration over all ${\theta }_j$’s in the real case is the following: for $\Re \left(\gamma \right)>-\frac{q}{2}+\frac{p-1}{2},\Re \left(\eta \right)>0,\delta >0,\alpha >0,p\le q,M=E\left[X\right]$,and $A>O,B>O$ are constant matrices where A is $p\times p$ and B is $q\times q$ and X is $p\times q,p\le q$ real matrix of rank p, and the corresponding integral in the complex case is given by for $\Re \left(\gamma \right)>-q+p-1,\Re \left(\eta \right)>0,\delta >0,\alpha >0,\tilde{M}=E\left[\tilde{X}\right],A=A^{*}>O,B=B^{*}>O,p\le q$. Therefore the normalizing constants c and $\tilde{c}$ are the following: for $\delta >0,\alpha >0,\Re \left(\eta \right)>0,p\le q$ and for $\delta >0,\alpha >0,\Re \left(\eta \right)>0,p\le q$.