As is widely known, nonlinear system control is an important topic of control fields, especially for uncertainly unknown nonlinear systems, which is difficult for traditional control methods. Until 1988, radial basis function neural networks were proposed [1]. Immediately following in 1900, Narendra, K. S. and K. Parthasarathy first proposed an artificial neural network adaptive control method for nonlinear dynamical systems [2]. Since then, multilayer neural networks (MNN) and radial basis function (RBF) neural networks were successfully applied in pattern recognition and control systems [3]. Compared to multilayer feedforward networks (MFNs), the RBF neural networks attracted much attention due to their good generalization ability, simple network structure, and avoidance of unnecessary and lengthy computations. Studies on RBF-NNs have shown the ability of neural networks to approximate any nonlinear function with a compact set and arbitrary accuracy[4,5]. Many research results have been published on neural network control for nonlinear systems [6,7].

On the other hand, optimal tracking control as one of the effective methods for nonlinear systems in optimal control, received many practical engineering applications [8,9,10]. Therefore, exploring the optimal tracking optimal control for nonlinear systems possesses significant theoretical importance and practical value. For optimal control methods for nonlinear systems, the difficulty lies in the requirement of solving the nonlinear Hamilton-Jacobi-Belman (HJB) equation, which is usually difficult to solve analytically. Although dynamic programming is an effective method for solving optimal control problems, there is the problem of "curse of dimensionality" when facing relatively complex systems [11,12].

Faced with the difficult problem of solving nonlinear Hamilton-Jacobi-Bellman partial differential equations exactly, several methods was proposed to approximate the solutions of the Hamilton-Jacobi-Bellman equations, These include the use of reinforcement learning [8,13,14,15,16,17,18,19] and back-propagation through time [20]. Among these classical RL methods, combining the advantages of adaptive control and optimal control, the ADP algorithm was considered as one of the core methods for realizing optimal control strategies for the diversity of optimal control problems, and it has been successfully applied to both continuous-time systems [21,22,23] and discrete-time systems [24,25,26,27,28] to search for solutions of the HJB equations online. Numerous ADP and RL approaches emerged, such as robust ADP [29,30] iterative/invariant ADP [31,32,33], spiking/Hamiltionian-driven ADP [34,35], integral RL [36,37], and off-policy RL [38,39,40]. Several works have attempted to solve the discrete time nonlinear optimal regulation problem in a near optimal sense using adaptive dynamic programming through neural networks (NNs) with offline training.

In the past decades, many relevant studies was conducted on the optimal tracking control of discrete-time nonlinear system, such as generalised policy iteration adaptive dynamic programming[41], actor-critic algorithm [42], heuristic dynamic programming (HDP)[43], greedy heuristic dynamic programming iteration algorithm[44] and Q-Learning Algorithm[45]. However, in the known literatures, optimal tracking control methods using RBF-NNs applied to the ADP algorithm are barely used.

In this paper, an optimal tracking control method RBF-NNs-based for discrete-time partially unknown nonlinear systems is proposed, two RBF neural networks are used to approximate the unknown system dynamic as well as the steady-state control, and after transforming the tracking problem into a regulation problem, two feedforward neural networks are used to approximate the critic network and the actor network to obtain the error feedback control, which allows the online learning process to require only current and past system data rather than the exact system dynamics.

The contributions of article are as follows: (1) Unlike classical technique of NNs approximating [42,44,45,46], we propose an near-optimal tracking control scheme for a class of partially unknown discrete-time nonlinear systems based on RBF-NNs and the stability of systems is proved by the Lyapunov theory. (2) Compared with [41,44], we additionally used an RBF-NN to directly approximate the the steady-state controller of the unknown system, it can solve the requirement for the priori knowledge of the controlled system dynamics and reference system dynamics. (3) For the inverse dynamic NN to directly approximate the steady-state controller of the system, we propose an novel adaptive law to update the weight of the RBF-NN, and the convergence is completed through the selection of constants.

The organization of this paper is as follows. The problem statement is shown in Section II. The technique of the system with partially unknown nonlinear dynamic are designed in Section III, where include the RBF-NN identifier, the RBF-NN steady-state controller, near optimal feedback controller and stability analysis. Simulations and experimental results are provided in Sections IV to validate the proposed control method. Section V draws some conclusions.

In this paper, we consider the following discrete-time nonlinear system: where $x\left(k\right)\in {\mathbb{R}}^n$ is the measurable system state and $u\left(k\right)\in {\mathbb{R}}^m$ is the control input. Assume that the nonlinear smooth function $f\left[x\right(k\left)\right]\in {\mathbb{R}}^n$ is an unknown drift function, $g\left[x\right(k\left)\right]\in {\mathbb{R}}^{n\text{×}m}$ is a known function and ${\parallel g\left[x\left(k\right)\right]\parallel }_F\le g_1$ where the Frobenius norm ${\parallel ·\parallel }_F$ is applied. In addition, assuming that there exists a matrix ${g\left[x\right(k\left)\right]}^{+}\in {\mathbb{R}}^{m\text{×}n}$ such that $g\left[x\left(k\right)\right]{g\left[x\right(k\left)\right]}^{+}=I\in {\mathbb{R}}^{n\text{×}n}$ where I is the identity matrix. Let $x\left(0\right)$ be the initial state.

The reference trajectory is generated by the following bounded command: where $x_d\left(k\right)$$\in {\mathbb{R}}^n$ and $\phi \left(x_d\left(k\right)\right)\in {\mathbb{R}}^n$, and $x_d\left(k\right)$ is the reference trajectory, which needs only to be stable in the sense of Lyapunov, not necessarily asymptotically stable.

Let $u\left(k\right)$ be an arbitrary sequence of controls from k to infinity. The goal of this paper is to design a controller $u\left(k\right)$ that not only ensures the state of system (1) tracks the reference trajectory, but also minimizes the cost function where $Q\in {\mathbb{R}}^{n\text{×}n}$ and $R\in {\mathbb{R}}^{m\text{×}m}$ are symmetric positive definite; $e\left(k\right)=x\left(k\right)-x_d\left(k\right)$ is tracking error. For common solutions of tracking problems [47], the control input consists of two parts, a steady-state input $u_d$ and a feedback input $u_e$. Next, we will discuss how obtain each part.

The steady-state of the control input is used to ensure perfect tracking. This perfect tracking equation is realized under the condition $x\left(k\right)=x_d\left(k\right)$. For this condition to be fulfilled, the steady-state part of the control $u_d\left(k\right)$ must exist to make $x\left(k\right)$ equivalent $x_d\left(k\right)$. By substituting $x_d\left(k\right)$ and $u_d\left(k\right)$ into system (1), the reference state is

If the system dynamics (1) are known, $u_d\left(k\right)$ is acquired by where $g{\left[x_d\left(k\right)\right]}^{+}={\left(g{\left[x_d\left(k\right)\right]}^{T}g\left[x_d\left(k\right)\right]\right)}^{-1}g{\left[x_d\left(k\right)\right]}^T$ is the generalized inverse of $g\left[x_d\left(k\right)\right]$ with $g{\left[x_d\left(k\right)\right]}^{+}g\left[x_d\left(k\right)\right]=I$.

By using (1) and (4), the tracking error dynamics $e\left(k\right)$ are given by where $f_e\left(k\right)=g(e\left(k\right)+x_d\left(k\right))g{\left(x_d\left(k\right)\right)}^{+}(\phi \left(x_d\left(k\right)\right)-f\left(x_d\left(k\right)\right))+f(e\left(k\right)+x_d\left(k\right))-\phi \left(x_d\left(k\right)\right)$, $u_e\left(k\right)=u\left(k\right)-u_d\left(k\right)$ and $g_e\left(k\right)=g[x_d\left(k\right)+e\left(k\right)]$. $u_e\left(k\right)$$\in {\mathbb{R}}^m$ is the feedback control input. By minimizing the cost function, it is designed to stabilize the tracking error dynamics. For $e\left(k\right)$ under the control sequence, the cost function is defined as where $r\left(k\right)=e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)$, and $J_e(e\left(k\right),u_e\left(k\right))>0$ for $\forall e\left(k\right),u_e\left(k\right)\ne 0$. $Q\in {\mathbb{R}}^{n\text{×}n}$ and $R\in {\mathbb{R}}^{m\text{×}m}$ are symmetric positive definite, $x_d\left(k\right)$ and $x_d(k+1)$ are bounded to be tracked by the reference trajectory. The tracking error $e\left(k\right)$ is used in this study of the cost function of the optimal tracking control problem. This feedback control $u_e\left(k\right)$ is found by minimizing (7) to solve the extremum condition in the optimal control framework[8]. This result is

Then the standard control input is obtained where $u_d\left(k\right)$ is obtained from (5), $u_e^{*}\left(k\right)$ is obtained from (8).

The main results of this paper are based on the following definitions and assumptions.

In this section, firstly, we use an RBF-NN to approximate the unknown system dynamics $f\left[x\right(k\left)\right]$, and use another RBF-NN to approximate the steady-state controller $u_d\left(k\right)$. Secondly, two feedback neural networks are introduced to approximate the cost function and the optimal feedback control $u_e\left(k\right)$. Finally, the system stability is proved by selecting an appropriate Lyapunov function.