In this paper, we study the Brezis-Nirenberg problem on H-type groups $G\subset {\mathbb{R}}^m\times {\mathbb{R}}^n$ where $\mathsf{\Omega }\subset G$ is a smooth bounded subset with $C^2$ boundary and $-{\Delta }_G$ is the Kohn Laplacian. Here $m\ge 2$ is an even number and $Q=m+2n$ is the homogeneous dimension of G, $Q^{*}=\frac{2Q}{Q-2}$ is the critical exponent in the sense of the $L^2$ Folland-Stein inequality.

The classical Brezis-Nirenberg problem set on Entire space ${\mathbb{R}}^n$ is a model-based problems of the classical Yamabe problem (prescribed curvature problem) related to the Riemann manifolds, see Yamabe [30]. It says that for a Riemannian manifold $(M,g)$, finding a conformal metric $\tilde{g}$ such that the scalar curvature $R_{\tilde{g}}$ is a constant. If $\tilde{g}=u^{\frac{4}{m-1}}g$ is a new metric conformal to g, the scalar curvature $R_{\tilde{g}}$ of $\tilde{g}$ is given by If $R_{\tilde{g}}=\mu$ is a constant, then the problem is equivalent to finding a positive $C^{\infty }$ solution u to the Yamabe equation: This problem has the following nice variational formulation. Consider the constrained variational problem Then the Yamabe problem becomes the following problems:

(a) $\mu \left(M\right)$ depends only on the conformal class of g.

(b) $\mu \left(M\right)\le \mu \left(S^m\right)$, in which the sphere $S^m$ has the standard metric.

(c) If $\mu \left(M\right)<\mu \left(S^m\right)$, then the infimum is attained by a positive solution. Thus the metric $\tilde{g}=u^{\frac{4}{n-1}}g$ has constant scalar curvature $\mu \left(M\right)$.

Thanks to the contributions of Aubin [1] and Schoen [25], these issues have been completely resolved. The proof of (a) consists of the fundamental observation that problem (3) is conformally invariant. The proof of (b) begins with a thorough understanding of the special case of the sphere ${\mathbb{S}}^m$ in ${\mathbb{R}}^{m+1}$: The conformal change of variables given by stereographic projection converts the variational problem on ${\mathbb{S}}^m$ to the more familiar problem on ${\mathbb{R}}^{m+1}$: In other words, by a stereographic projection, $\mu \left(S^m\right)=\mu \left({\mathbb{R}}^m\right)$. While $\mu \left({\mathbb{R}}^m\right)$ is the sharp constant of the Sobolev inequality on ${\mathbb{R}}^{m+1}$: and it is attained by a Talenti bubble.

Therefore, (c) can be easily proved by transplanting an approximate extremal function from ${\mathbb{R}}^{m+1}$ to a small neighborhood on M.

Actually, when further examining the apparent solution, Trudinger [27] pointed out the problem of lacking compactness. Therefore, Briezs and Nirenberg [4] proposed a detection of the Yamabe equation from an analytical perspective involves constructing model-based problems: where $\mathsf{\Omega }\subset {\mathbb{R}}^m$ is a bounded domain and $2^{*}=\frac{2m}{m-2}$ is the Sobolev critical exponent. This problem has been solved in different domains, see [6,7,13].

As we are aware, similarly to how Riemann manifolds function in Classical Mechanics, the Heisenberg group plays a significant role in Quantum Mechanics due to its noncommutative structure, see [5]. Additionally, the Heisenberg group is considered as a Cauchy-Riemann (call CR short) manifold, intersecting complex analysis, differential geometry, and the theory of partial differential equations, see [10]. On the other hand, since Wells [29] established the CR structure on complex manifold, Chern, Moser [8], Webster [28] and Tanaka[26] found a complete system of analytic invariant for two real analytic real hypersurfaces in complex manifold to be locally equivalent under biholomorphic transformation, one then can define the unique connect form $\theta$ of any CR manifold, and hence, the Webster scalar curvature $R_{\theta }$. Therefore, there are far-reaching analogies between conformal and CR geometries, see [20]. The CR-Yamabe problem says that for a CR manifold $(N,\theta )$, finding a contact form $\tilde{\theta }$ such that the Webster scalar curvature $R_{\tilde{\theta }}$ is a constant. If $\tilde{\theta }=u^{\frac{4}{n-1}}\theta$ is a new contact form conformal to $\theta$, the Webster scalar curvature $R_{\tilde{\theta }}$ of $\tilde{\theta }$ is given by If $R_{\tilde{\theta }}=\lambda$ is a constant, then the problem is equivalent to finding a positive $C^{\infty }$ solution u to the CR-Yamabe equation: Consider the constrained variational functional Then the CR-Yamabe problem becomes the following problems:

(d) $\lambda \left(N\right)$ depends only on the conformal class of $\theta$.

(e) $\lambda \left(N\right)\le \lambda \left(S^{2n+1}\right)$, in which the sphere $S^{2n+1}$ has the standard CR structure.

(f) If $\lambda \left(N\right)<\lambda \left(S^{2n+1}\right)$, then the infimum is attained by a positive solution. Thus the metric $\tilde{\theta }=u^{\frac{4}{n-1}}\theta$ has constant Webster scalar curvature $\lambda \left(N\right)$.

The proof of (d-f) is quite similar to the classical case. By the Cayley transform, the variational problem on $S^{2n+1}$ can be transformed into the variational problem on ${\mathbb{H}}^n$ This relates to the Folland-Stein inequality on ${\mathbb{H}}^n$ Building on Obata’s proof [24], Jerison and Lee [21] demonstrated the uniqueness of positive optimizers by deriving a Bianchi identity for the CR function that undergoes a conformal transformation. As a model-based problems, Citti [9] also consider the Brezis-Nirenber problem on ${\mathbb{H}}^n$ where $\mathsf{\Omega }\subset {\mathbb{H}}^n$ is a bounded domain, $Q=2n+2$ is the homogeneous dimension of ${\mathbb{H}}^n$ and $Q^{*}=\frac{2Q}{Q-2}$ is the critical exponent of Folland-Stein inequality. Readers can refer [16,18] to the nonlocal case.

In the 1980s, Kaplan [22] introduced an H-type group in connection with hypoellipticity questions, showing similarities with the Heisenberg group but with more intricate properties, see Section 2. Particularly, in paper [22], Kaplan also established the similar Folland-Stein inequality on H-type group G where $\mathrm{d}g$ is the Haar measure on G. Following this result, Han and Niu [19] established the corresponding Hardy-Sobolev type inequalities on H-type group. Interested readers can refer to [33] for the Caffarelli-Kohn-Nirenberg inequality, and to [23] for the Stein–Tomas type inequality on H-type group.

For further insights on the H-type group, Garofalo and Vassilev [14,15] presented the Yamabe-type problem concerning this group. Strangely, since Kaplan [22] determined the explicit form of the optimizers for (13), the uniqueness of the optimizers has been a longstanding question. Nearly 20 years later, Garofalo and Vassilev [14,15] provided a definitive answer when the H-type group is an Iwasawa-type group. Recently, Yang [31] eventually proved the uniqueness of the optimizer using a sub-critical approximate methods. However, due to the absence of Cayley transform on the general H-type, proving the the non-degeneracy of the optimizer become challenging. As a result, extending the perturbation result of the Yamabe equation from Iwasawa-type groups (see [32]) to H-type Groups remains unfeasible. This situation prompts us to examine the linear perturbation problem of the critical equation on the bounded domain, that is the Brezis-Nirenberg problem (1) on the H-type group. As we can see, the solution to the Brezis-Nirenberg problem greatly depends on the dimension and topological properties of the domain, as highlighted in [6,7]. Therefore, researching the Brezis-Nirenberg problem on an H-type group with non-homogeneous dimension holds considerable significance. By carefully analyzing and utilizing the variational method, especially the Linking argument, the primary outcome is as follows:

Turing to layout of the paper, in Section 2, we give some preliminaries knowledge on H-type group. For the sake of clarity, we establish the variational frame and make some compactness analysis in Section 3. The total proof has been divided into two cases, one is $0<\lambda <{\lambda }_1$ and another is $\lambda >{\lambda }_1$. All of these process are completed in Section 4. Particularly, we prove the first case by a mountain-pass theorem and we prove the second case by a Linking argument.

Let’s begin with some basics of Carnot group. Carnot group (in a short $CG$) is a stratified, simply connected nilpotent Lie group of step r. Denote by $\mathcal{G}$ be Lie algebre of $CG$. It is known that $\mathcal{G}={\oplus }_{i=1}^r{V}_i$ satisfying (see e.e. [12]) As a simply connected nipotent group, $CG$ is differential with ${\mathbb{R}}^N$, $N={\displaystyle \sum _{i=1}^r}\dim V_i$, via the exponential map $\exp :\mathcal{G}\to CG$. There is a natural family of nonisotropic dilations ${\delta }_{\lambda }:\mathcal{G}\to \mathcal{G}$ for $\lambda >0$ and we define it as follow: The homogeneous dimension of $CG$, associated with ${\delta }_{\lambda }$, is $Q={\displaystyle \sum _{j=1}^r}j\dim V_j$.

An H-type group G is a Carnot group with the following properties (see Kaplan [22]): the Lie algebra $\mathcal{G}$ of G is endowed with an inner product $\langle ·,·\rangle$ such that, if $\mathcal{Z}$ is the center of $\mathcal{G}$, then $[{\mathcal{Z}}^{\perp },{\mathcal{Z}}^{\perp }]=\mathcal{Z}$ and moreover, for every fixed $z\in \mathcal{Z}$, the map $J_z:={\mathcal{Z}}^{\perp }\to {\mathcal{Z}}^{\perp }$ defined by is an orthogonal map whenever $\langle z,z\rangle =1$. Set $m=\dim {\mathcal{Z}}^{\perp }$ and $n=\dim \mathcal{Z}$. Since G has step two, we can fix on G a system of coordinates $(x,t)$ such that the group law on G has the form (see [2]) for suitable skew-symmetric matrices $U^{\left(j\right)}$’s. Note that $U^{\left(j\right)}$ is also a orthogonal matrix, hence m has to be an even number, see [2]. The vector field in the Lie algebra $\mathcal{G}$ that agrees at the origin with $\frac{\partial }{\partial x_j}(j=1,···,m)$ is given by and $\mathcal{G}$ is spanned by the left-invariant vector fields $X_1,···,X_m$, $T_1=\frac{\partial }{\partial t_1},···,T_n=\frac{\partial }{\partial t_n}$. Furthermore (see [2], Page 200, (A.4)),

The horizontal gradient on G is ${\nabla }_G=(X_1,···,X_m)$. The sub-Laplacian on G is given by (see [2], Remark A.6.) where

The Sobolev space $W_0^{1,2}\left(G\right)$ is the closure of $C_0^{\infty }\left(G\right)$ with respect to the norm where $\mathrm{d}g$ is the Haar measure on G. We remark that the Haar measure on G, induced by the exponential mapping from the Lebesgue measure on $\mathcal{G}={\mathbb{R}}^N$, coincides the Lebesgue measure on ${\mathbb{R}}^N$. Correspondingly, in a bounded domain $\mathsf{\Omega }\subset G$, we define the Sobolev space as $W_{0,G}^{1,2}\left(\mathsf{\Omega }\right)$.

The Folland-Stein inequality on G reads that there exist some constant C such that for each $u\in W_0^{1,p}\left(G\right)$ (see [11,20,21]) More precisely, Yang [31] obtained the sharp constant $S_{m,n}=C$ and obtained the uniqueness of the extremal unction $U\left(\xi \right)$. Namely, set it has been shown that ${\left[\frac{m(Q-2)}{4}\right]}^{\frac{Q-2}{4}}{U}_{\lambda ,\eta }\left(\xi \right)$ satisfies the Yamabe-type equation (see [14,15]) or equivalently,

Based on the group law (14), the nonisotropic dilations ${\delta }_{\lambda }$ on G is For $(x,t)\in G$, the homogeneous norm of $(x,t)$ is With this norm $\rho$, we can define the ball centered at origin with radius R

In order to study the problem by variational methods, we introduce the energy functional associated to (1) by

Firstly, we check that the energy functional $J_{\lambda }\left(u\right)$ has a mountain structure as $\lambda \in (0,{\lambda }_1)$.

As $\lambda \in [{\lambda }_j,{\lambda }_{j+1})$ for some $j\in \mathbb{N}$, we then easily check that $J_{\lambda }$ satisfies a Link structure. Set ${\left\{e_j\right\}}_{j\in \mathbb{N}}\subset L^{\infty }\left(\mathsf{\Omega }\right)$ be the sequence of eigenfunctions corresponding to ${\lambda }_j$. We know that this sequence is an orthonormal basis of $L^2\left(\mathsf{\Omega }\right)$ and the orthogonal basis of $W_{0,G}^{1,2}\left(\mathsf{\Omega }\right)$. We denote and denote ${\mathrm{Y}}_j:=\mathrm{span}\{e_1,···,e_j\}$ by the linear subspace generated by the first j eigenfunctions of $-{\Delta }_G$ for any $j\in \mathbb{N}$. Note that ${\mathbb{Y}}_j$ is finite dimensional and ${\mathbb{Y}}_j\oplus {\mathbb{E}}_{j+1}=W_{0,G}^{1,2}\left(\mathsf{\Omega }\right)$, then $J_{\lambda }$ satisfies the following properties:

Similar to Proposition 1, we have the next proposition for $\lambda \in [{\lambda }_j,{\lambda }_{j+1}]$.

Now, we are discuss the boundedness of these $\left(PS\right)$ sequence.

To make sure the weak solution $u_0\in W_{0,G}^{1,2}\left(\mathsf{\Omega }\right)$, we shall prove the $\left(PS\right)$ sequence has a strong convergence subsequence by controlling the energy of the functionals. Before this, we need a Brezis-Lieb lemma to describe the properties of the $\left(PS\right)$ sequence.

Based on this Lemma, we have the following claim.

To control $c^{*}<\frac{1}{Q}{S}_{m,n}^{\frac{Q}{2}}$ and $c^{★}<\frac{1}{Q}{S}_{m,n}^{\frac{Q}{2}}$ by some $\lambda$, we recall some invariant properties of extremal function for the sharp constant $S_{m,n}$. Since the functional associated the Sobolev inequality is invariant under dilation, define we then have Observe that the critical point of $c^{*}$ or $c^{★}$ is vary closed to the extermal function, we take a cut-off method to estimate every term of $J_{\lambda }\left(u\right)$. Without loss of generality, we may assume that $0\in \mathsf{\Omega }$ and $B_R\subset \mathsf{\Omega }\subset B_{2R}$ for some positive R. Let $\psi \in C_0$ such that and $0\le \psi \left(x\right)\le 1$, $|{\nabla }_G\psi \left(x\right)|\le C$ for all $x\in G$. Define we then have the following estimate: