1. Introduction

2. Problem Statement

3. Method

3.2. Soft Actor Critic

Although the DQN algorithm, which is a milestone in deep reinforcement learning, solves the problem of high-dimensional and continuous input states that cannot be solved by classical reinforcement learning, it still cannot solve the situation where the output actions are high-dimensional and continuous (such as multi-degree-of-freedom manipulator). Although the other deep reinforcement learning algorithms (such as DDPG [23], TD3 [24], and other algorithms) can handle the case where the output action is high dimensional and continuous, they usually have a high sample complexity and weak sample convergence, which lead to some additional hyperparameter tuning.

The Soft Actor-Critic (SAC) algorithm [25] is a reinforcement learning algorithm that introduces the maximum entropy theory. In the framework of the algorithm, the strategy not only needs to maximize the expected cumulative reward value, but also needs to maximize the expected entropy:

$${\pi }^{*}=\max {\displaystyle \sum _{t=0}^T{E}_{(s_t,a_t)~p_{\pi }}[r(s_t,a_t)+\alpha H(\pi (\cdot |s_t))]}$$

(5)

where $\mathsf{\alpha }$ is the weight of the entropy term, which can determine the relative importance of the entropy term relative to the reward term, thereby controlling the randomness of the optimal strategy.

In order to maximize this goal, a method of alternating policy evaluation and policy improvement is used in the maximum entropy framework - Soft Policy Iteration. In an environment where the state space is discrete, the method can obtain the soft Q-value from the randomly initialized function $Q:S\times A\to R$ and repeatedly apply the modified Bellman backup operator $T^{\pi }$ :

$$T^{\pi }Q(s_t,a_t)=r(s_t,a_t)+\gamma E_{s_{t+1\sim p}}[V(s_{t+1})]$$

(6)

where

$$V(s_t)=E_{a_t~\pi }[Q(s_t,a_t)-\log \pi (a_t|s_t)]$$

(7)

is the soft state value function used to calculate the policy value in policy evaluation. While in the continuous state, a neural network with parameters is first used to replace the soft Q-function $Q_{\theta }\left(s_t, a_t\right)$, and then it is trained to minimize the Bellman residual:

$$J_Q(\theta )=E_{(s_t,a_t)~D}\left[\frac{1}{2}{\left(Q_{\theta }(s_t,a_t)-\left(r(s_t,a_t)+\gamma E_{s_{t+1}~p}[V_{\overline{\theta }}(s_{t+1})]\right)\right)}^2\right]$$

(8)

which can also be optimized with stochastic gradients:

$${\stackrel{\wedge }{{\displaystyle \nabla }}}_{\theta }{J}_Q(\theta )={{\displaystyle \nabla }}_{\theta }{{\displaystyle Q}}_{\theta }(s_t,a_t)({{\displaystyle Q}}_{\theta }(s_t,a_t)-r(s_t,a_t)-\gamma {{\displaystyle V}}_{\overline{\theta }}(s_{t+1}))$$

(9)

where $V_{\overline{\theta }}\left(s_{t+1}\right)$ is estimated by the target network of Q and the Monte Carlo estimation of the soft state value function sampled from the experience pool.

Policy improvement is updating the policy in the direction of maximizing its available reward, therefore, the policy needs to be updated to the exponential form of the new soft Q-function and restricted to some parameterized distribution (such as Gaussian distribution) and then project it back into the acceptable policy space using an information projection defined in terms of the Kullback-Leibler (KL) divergence.

$${\pi }_{new}=\arg \underset{{\pi }^{\prime }\in \Pi }{\min }{D}_{KL}\left({\pi }^{\prime }(\cdot |s_t)\parallel \frac{\exp (Q^{{\pi }_{old}}(s_t,\cdot ))}{Z^{{\pi }_{old}}(s_t)}\right)$$

(10)

where $Z^{{\pi }_{old}}\left(s_t\right)$ can be ignored because it has no effect on the gradient. And parameterize the policy ${\pi }_{\varphi }\left(a_t|s_t\right)$ with a neural network that can output mean and variance to define a Gaussian distribution, and then learn the parameters of the policy by minimizing the expected KL divergence:

$$J_{\pi }\left(\varphi \right)=E_{s_t~D}\left[E_{a_t~{\pi }_{\varphi }}\left[\log {\pi }_{\varphi }(a_t|s_t)-Q_{\theta }(s_t,a_t)\right]\right]$$

(11)

However, since the Gaussian distribution $a~N\left(m,s\right)$ is difficult to find its gradient, it is converted into an easy-to-find gradient form $a=m+s\epsilon$, $\epsilon ~N\left(0, 1\right)$, i.e. $a_t=f_{\varphi }\left({\epsilon }_t,s_t\right)$, the policy network can then be optimized by applying the policy gradient to the expected future reward:

$$J_{\pi }\left(\varphi \right)=E_{s_t~D,{\epsilon }_t~N}\left[\log {\pi }_{\varphi }(f_{\varphi }({\epsilon }_t;s_t)|s_t)-Q_{\theta }(s_t,f_{\varphi }({\epsilon }_t;s_t))\right]$$

(12)

We can also approximate the gradient of Equation (12) with:

(13)

4. Experiments and Results

5. Conclusion

References

1. Parlaktuna, O.; Ozkan, M. Adaptive control of free-floating space robots in Cartesian coordinates. Adv Robotics 2004, 18, 943–959. [Google Scholar] [CrossRef]
2. Guo, Y.; Chen, L. Adaptive neural network control for coordinated motion of a dual-arm space robot system with uncertain parameters. Appl Math Mech 2008, 29, 1131–1140. [Google Scholar] [CrossRef]
3. Canudas, W.C.; Fixot, N. Robot control via robust estimated state feedback. IEEE T Automat Contr 1991, 36, 1497–1501. [Google Scholar] [CrossRef]
4. Kim, E. ; Output feedback tracking control of robot manipulators with model uncertainty via adaptive fuzzy logic. IEEE T Fuzzy Syst 2004, 12, 368–378. [Google Scholar]
5. Abdollahi, F.; Talebi, H. A, Patel RV. A stable neural network-based observer with application to flexible-joint manipulators. IEEE Trans Neural Netw 2006, 17, 118–129. [Google Scholar] [CrossRef]
6. Kim, Y.H.; Lewis, F.L. Neural network output feedback control of robot manipulators. IEEE Trans Rob Autom 1999, 15, 301–309. [Google Scholar] [CrossRef]
7. Selmic, R.R.; Lewis, F.L. Dead zone compensation in motion control systems using neural networks. IEEE Trans Automat Contr. 2000, 45, 602–613. [Google Scholar] [CrossRef]
8. Zhu, W.H.; Lamarche, T. Dupuis, E. et al. Networked embedded control of modular robot manipulators using VDC. IFAC Proc 2014, 47, 8481–8486. [Google Scholar] [CrossRef]
9. Cao, S.; Jin, Y.; Trautmann, T.; Liu, K. Design and Experiments of Autonomous Path Tracking Based on Dead Reckoning. Appl. Sci. 2023, 13, 317. [Google Scholar] [CrossRef]
10. Cai, Z.X.; Robotics. Tsinghua university press: Beijing (BJ), China, 2000.
11. Patolia, H.; Pathak, P.M.; Jain, S.C. Force control in single DOF dual arm cooperative space robot. P 2010 Spr Simul Multicon 2010, 1–8. [Google Scholar]
12. Ma, B.L.; Huo, W. Adaptive Control of Space Robot System. Iet Control Theory A 1996, 13, 191–197. [Google Scholar]
13. Spong, M.W. ;On the robust control of robot manipulators. Ieee T Automat Contr 1992, 37, 1782–1786. [Google Scholar] [CrossRef]
14. Purwar, S.; Kar, I.N.; Jha, A.N. Adaptive output feedback tracking control of robot manipulators using position measurements only. Expert Syst Appl 2008, 34, 2789–2798. [Google Scholar] [CrossRef]
15. Annusewicz-Mistal, A.; Pietrala, D.S.; Laski, P.A.; Zwierzchowski, J.; Borkowski, K.; Bracha, G.; Borycki, K.; Kostecki, S.; Wlodarczyk, D. Autonomous Manipulator of a Mobile Robot Based on a Vision System. Appl. Sci. 2023, 13, 439. [Google Scholar] [CrossRef]
16. Zhang, T.; Song, Y.; Kong, Z.; Guo, T.; Lopez-Benitez, M.; Lim, E.; Ma, F.; Yu, L. Mobile Robot Tracking with Deep Learning Models under the Specific Environments. Appl. Sci. 2023, 13, 273. [Google Scholar] [CrossRef]
17. Tappe, S.; Pohlmann, J.; Kotlarski, J. et al. Towards a follow-the-leader control for a binary actuated hyper-redundant manipulator. In Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 28 September–2 October 2015; pp. 3195–3201.
18. Palmer, D.; Cobos-Guzman, S.; Axinte, D. Real-time method for tip following navigation of continuum snake arm robots. Robot Auton Syst 2014, 62, 1478–1485. [Google Scholar] [CrossRef]
19. Sutton, R.S.; Barto, A.G. Reinforcement learning: An introduction; MIT press: Cambridge (MA), Britain, 2018. [Google Scholar]
20. Guo, X. Research on the control strategy of manipulator based on DQN. master’s thesis, Beijing Jiaotong University, Beijing(BJ), China, 2018. [Google Scholar]
21. Liu, Y.C.; Huang, C.Y. DDPG-Based Adaptive Robust Tracking Control for Aerial Manipulators With Decoupling Approach. IEEE Trans Cybern 2022, 52, 8258–8271. [Google Scholar] [CrossRef]
22. Mnih, V.; Kavukcuoglu, K.; Silver, D. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef]
23. Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.et al. Continuous control with deep reinforcement learning. ArXiv [Preprint] 2015, arXiv:1509.02971.
24. Fujimoto, S.; Hoof, H.; Meger, D. Addressing function approximation error in actor-critic methods. International conference on machine learning 2018, PMLR, 1587–1596. [Google Scholar]
25. Haarnoja, T.; Zhou, A.; Abbeel, P. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning 2018, 1861–1870. [Google Scholar]
26. LaValle, SM. Rapidly-exploring random trees: A new tool for path planning. 1999, Research Report.