1. Introduction

2. Related Work

3. Proposed Method

3.3. Hierarchical Policy

In this section, we utilize HRL to complete path-planning tasks under the guidance of leveraging visual cues for the action selection model. A two-level policy layer is conducted to increase efficiency without sacrificing exploring capability.

(1) Environment representation

The environments are represented in two forms, e.g. high-level grid form and low-level ones. The high-level network is designed to achieve global decision-making based on above work, while the lower-level is used for better local planning.

High-level grid map: It selects the next macro-action, directing the agent towards a specific region on the grid. The grid map is divided into larger blocks, and each block is treated as a high-level state. The high-level policy determines the sequence of blocks to be explored based on the current state and the goal location. The settings of the high-level grid help agents avoid areas where visual information has not been fully detected and accelerate the training process in the task of robotics path planning.

Figure 4. Data flow of the hierarchical policy.

Figure 4. Data flow of the hierarchical policy.

Low-level grid map: Within each selected high-level block, the low-level strategy navigates through the individual grid cells. The policy network produces action instructions for robotics when visual and state information is obtained in grid cells.

$$s_{low}\in \{{\mathrm{cell}}_{1,1},{\mathrm{cell}}_{1,2},\dots ,{\mathrm{cell}}_{m,m}\}$$

(7)

(2) RL structure

We model after the design of DQN and incorporate this structure into the proposed framework: a policy network and a target network. As shown in Figure 5, the policy network is updated continuously based on the Bellman equation and the target network and generates policies for explored areas ${\pi }_{explored}$. APF is applied to generate policies for unexplored areas ${\pi }_{unexplored}$. ${\pi }_{explored}$ and ${\pi }_{unexplored}$ are combined to form policy ${\pi }_{global}$ for the target net. The target network’s parameters are updated less frequently, specifically, every 100 episodes, by comparing the global strategy with the most recent policy network.

$$Q(s,a;\theta )\to \mathrm{policy}\mathrm{network}$$

(8)

$$Q^{\prime }(s,a;{\theta }^{-})\to \mathrm{target}\mathrm{network}$$

(9)

The action selection is based on the $\epsilon$-greedy policy, balancing exploration and exploitation. The Q-value updates follow the standard Bellman equation:

$$Q(s_t,a_t)=r_t+\gamma \underset{a^{\prime }}{max}{Q}^{\prime }(s_{t+1},a^{\prime };{\theta }^{-})$$

(10)

where $r_t$ is the reward at time t, $\gamma$ is the discount factor, and ${\theta }^{-}$ are the parameters of the target network.

Additionally, the policy network parameters are updated based on the target network to ensure stability:

$$\theta \leftarrow \theta +\alpha \left(y-Q(s,a;\theta )\right){\nabla }_{\theta }Q(s,a;\theta )$$

(11)

where $y=r_t+\gamma {max}_{a^{\prime }}{Q}^{\prime }(s_{t+1},a^{\prime };{\theta }^{-})$.

(3) Base policy and reward settings

Visual servoing: For areas already explored, we employ the visual-act model to map input images to action directives. The network processes the visual input and outputs an action for the agent. During every step, these action instructions are added to a policy pool, which serves as the optimal policy to determine the parameters of the target net.

$$a_{(x,y)}=\pi (s_{low}\mid I_{(x,y)})$$

(12)

$${\pi }_{explored}\leftarrow \bigcup _{(x,y)\in \mathbf{E}}^n{a}_{(x,y)}$$

(13)

where $a_{(x,y)}$ is the action at the coordination $(x,y)$, $I_{(x,y)}$ is the visual input, and $\pi$ is the policy function parameterized by the pre-trained visual-act decision model. ${\pi }_{explored}$ is the policy developed by the pre-trained model for the explored area.

Artificial potential field for unexplored regions: For high-level blocks that have not been explored, we utilize the artificial potential field method to generate navigation strategies. The APF method uses an attractive potential to pull the agent towards the goal and a repulsive potential to avoid obstacles. The direction generated by the APF is then decomposed to produce specific actions. The potential field is represented as U, which can be decomposed into two parts, $U_{att}$ and $U_{rep}$. The formula is represented as below:

$$U_{att}={\displaystyle \frac{1}{2}}{k}_{att}{d}^2$$

(14)

$$U_{rep}=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{k}_{rep}{\left({\displaystyle \frac{1}{d_o}}-{\displaystyle \frac{1}{d_{o0}}}\right)}^2\hfill & \mathrm{if}{d}_o\le d_{o0}\hfill \\ 0\hfill & \mathrm{if}{d}_o>d_{o0}\hfill \end{array}\right.$$

(15)

where $k_{att}$ and $k_{rep}$ are constants, d is the distance to the goal, $d_o$ is the distance to the obstacle, and $d_{o0}$ is the influence distance of the obstacle.

The combined potential field U determines the direction $\overrightarrow{F}$ for navigation:

$${\pi }_{unexplored}\leftarrow \overrightarrow{F}=-\nabla U$$

(16)

This direction is then translated into specific actions within the unexplored high-level blocks as the ${\pi }_{unexplored}$. The whole process is conducted after every episode is done.

Global strategy: It is synthesized by combining the policies derived from both the explored and unexplored regions. The synthesized strategy is then compared with the historical strategy obtained from past episodes. The target network parameters are updated based on the comparison to ensure the new strategy optimizes over the past strategies.

$${\pi }_{global}={\pi }_{explored}\cup {\pi }_{unexplored}$$

(17)

Rewards settings: The reward functions are listed below. The $s_{next}$ represents the next state of the agent. The $s_{target}$ represents the target state. The $R_{collision}$, $R_{safe}$ represent the reward value the agent receives in collision condition and safe condition respectively.

1) target distance

$$R_1(s,a)=-\parallel s_{next}-s_{target}\parallel$$

(18)

2) obstacle avoidance

$$R_2(s,a)=\left\{\begin{array}{cc}{R}_{collision}\hfill & \mathrm{if}\mathrm{collision}\hfill \\ R_{safe}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(19)

3) whether the agent reach the target

$$R_3(s,a)=\left\{\begin{array}{cc}{R}_{target}\hfill & \mathrm{if}{s}_{next}=s_{target}\hfill \\ R_{step}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(20)

(4) Adaptive experience replay management

The management of the experience replay buffer [28] is adaptively adjusted based on the cumulative average reward, which serves as an indicator of the learning progress and the efficiency of the current policy. The cumulative average reward, $R_{\mathrm{avg}}$, is computed as follows:

$$R_{\mathrm{avg}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{R}_i$$

(21)

where $R_i$ is the immediate reward received after the i-th action, $R_g$ represents rewards of achieving the goal, $R_o$ represents rewards of encountering obstacles, and N is the total number of actions taken up to the current point in time. If the agent achieves the goal or encounters obstacles, $R_g$ or $R_o$ is added to $R_{\mathrm{avg}}$ and suspend this episode instantly.

Based on $R_{\mathrm{avg}}$, we adjust the buffer size, $B_{\mathrm{size}}$, and the batch size, $b_{\mathrm{size}}$, of the experience replay buffer to enhance the learning efficiency. The adjustments are made according to the following rules:

$$B_{\mathrm{size}}=B_{\min }+(B_{\max }-B_{\min })·min\left({\displaystyle \frac{R_{\mathrm{avg}}}{R_{\mathrm{target}}}},1\right)$$

(22)

$$b_{\mathrm{size}}=b_{\min }+(b_{\max }-b_{\min })·min\left({\displaystyle \frac{R_{\mathrm{avg}}}{R_{\mathrm{target}}}},1\right)$$

(23)

where $B_{\min }$ and $B_{\max }$ represent the minimum and maximum buffer sizes, respectively; similarly, $b_{\min }$ and $b_{\max }$ denote the minimum and maximum batch sizes. $R_{\mathrm{target}}$ is a target average reward that indicates an optimal learning performance. These formulas ensure that as the average reward approaches the target, both the buffer size and the batch size increase, allowing the model to learn from a larger set of experiences. Conversely, if the performance drops, the model focuses on a smaller, potentially more relevant set of experiences to adjust its policy more rapidly.

4. Simulation

4.3. Assessment of the Visual-Act Model

In this part, we trained our model on several diverse environments and tested its performance on a separate set of unseen environments that follow different distributions with the train set. The training environments utilize cityscapes [29]. Labels are modified to fulfill the model input requirements. It includes various layouts, obstacle types, and goal locations in real road scenes.

As shown in Figure 5, observations of the loss curve revealed a contracting trend during the training process, indicating a consistent enhancement in model accuracy. The training process converges around 200 epochs because the zero-shot image encoder is perfectly pre-trained, which can well reflect the semantic and spatial information of objects in the image in the coding and accelerate the convergence speed in the proposed method training process.

Figure 5. Loss curve of the pre-trained model.

Figure 5. Loss curve of the pre-trained model.

The loss of the model training drops rapidly in Area I, but it picks up in Area III. We speculate that when setting the loss function, we treat the similarity using the sigmoid function to guarantee the artificial number in logarithm loss calculation to be positive, and the gradient is amplified during the loss backward process; thus, the fluctuation occurs.

As shown in Figure 6, the test environments are designed to differ significantly from the training ones regarding layout complexity and obstacle placements, ensuring a rigorous assessment of the model’s generalization ability. The virtual environment is primarily designed to imitate a wilderness setting, and its significant differences from the urban road image data in the training set and the inexperience of the pre-trained model in such a scenario further contribute to the rigorousness of the assessment.

We conduct a comparative simulation experiment to confirm the focused area of the visual-act model. $I_0$ represents the embeddings produced by the original image with CLIP. $I_1$, $I_2$, and $I_3$ represent the image embeddings processed by the visual-act model. The sim($I_0$) value represents the similarity between the image embeddings and semantic vector of certain objects.

In Table 3, data labeled in bold have the highest similarity to the corresponding object after processing with the attention mechanism in all directions. For example, as shown in Figure 7, the tree is in front of the robot, and the similarity comparison represents that the correspondent processed embeddings raise focus on this obstacle from 0.2607 to 0.4121 while decreasing focus in irrelevant directions. It shows that the pre-trained model focuses on certain obstacles related to correspondent action while disregarding the unrelated ones.

4.5. Performance Assessment under Test Environments

In this part, we conduct the generalization simulation under several environments modeled after the trained ones.

(1) Settings

The test environments differ from the train ones. As shown in Figure 11, they follow a different distribution. The tests are conducted without extra training within unseen test environments. The distinction between the train environments and test environments is listed below:

1) The starting and target points follow different distributions with the train environments. Specifically, as shown in Figure 11a–c, they all lie in different directions.

2) The patterns of the available paths are different from the train environments.

3) The distributions of obstacles are different.

(2) Metrics

In this experiment, we emphasize the guiding role of ZSL and visual action at a high level. Therefore, the analysis is conducted on a grid map. We selected success rate and collision rate as metrics. The specific results are shown in Table 4 and Figure 12.

(3) Outcome

Figure 12. Path in inexperienced environments.

Figure 12. Path in inexperienced environments.

In inexperienced environment tests, as shown in Table 4, the collision rate and success rate indicate that the proposed method has significantly reduced the collision rate, effectively lowering the collision rate of the intelligent agent in the path planning task. Additionally, a certain degree of enhancement is observed in the success rate. The paths planned under the corresponding samples are shown in Figure 14. Among environments (a), (b), and (c), the performance under environment (b) is not good enough cause the target area cannot be seen in the most of time during the process thus the designed visual-act can not perform well in success rate, but the collision rate remains at a low level, indicating that the proposed method has excellent performances in obstacle avoidance.

Table 4. Similarities comparison.

Table 4. Similarities comparison.

Environment	S1	C2	${\mathit{S}}^{\prime }$3	${\mathit{C}}^{\prime }$4
(a)	9.25%	8.53%	21.62%	2.64%
(b)	3.31%	14.72%	11.91%	3.12%
(c)	16.28%	7.34%	31.88%	0.65%

14.1cm ¹ S represents the average success rate under several inexperienced environments for the traditional HRL method. ²C represents the average collision rate under several inexperienced environments for the traditional HRL method. ³$S^{\prime }$ represents the average success rate under several inexperienced environments for the proposed method. ⁴$C^{\prime }$ represents the average collision rate under several inexperienced environments for the proposed method.

5. Conclusion

References

1. Duchoň, F. Path planning with modified a star algorithm for a mobile robot. Procedia engineering 2014, 96, 59–69. [Google Scholar] [CrossRef]
2. Luo, M.; Hou, X.; Yang, J. Surface optimal path planning using an extended Dijkstra algorithm. IEEE Access 2020, 8, 147827–147838. [Google Scholar] [CrossRef]
3. Kennedy, J.; Eberhart, R. Particle swarm optimization. Proceedings of international conference on neural networks, Perth, Australia, Nov 27-Dec 1 1995; pp. 1942–1948. [Google Scholar]
4. Shi, Y.; Eberhart, R. A modified particle swarm optimizer. Proceedings of IEEE international conference on evolutionary computation, Anchorage, USA, 4-9 May 1998; pp. 69–73. [Google Scholar]
5. Grisetti, G.; Kümmerle, R.; Burgard, W. A tutorial on graph-based SLAM. IEEE Intelligent transportation systems magazine 2010, 2, 31–43. [Google Scholar] [CrossRef]
6. Zang, S.; Ding, M.; Smith, D. The impact of adverse weather conditions on autonomous vehicles: How rain, snow, fog, and hail affect the performance of a self-driving car. IEEE vehicular technology magazine 2019, 14, 103–111. [Google Scholar] [CrossRef]
7. Wang, Y.; Zou, S. Policy gradient method for robust reinforcement learning. Proceedings of the international conference on machine learning 23484–23526.
8. Zhang, J.; Koppel, A.; Bedi, A. Variational policy gradient method for reinforcement learning with general utilities. Advances in neural information processing systems 2020, 33, 4572–4583. [Google Scholar]
9. Lampert, C.; Nickisch, H.; Harmeling, S. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, Miami, USA, 20-25 June 2009; pp. 951–958. [Google Scholar]
10. Wang, W.; Zheng, W.; Han, Y. A survey of zero-shot learning: Settings, methods, and applications. ACM transactions on intelligent systems and technology 2019, 10, 1–37. [Google Scholar] [CrossRef]
11. Zhang, L.; Xiang, T.; Gong, S. Learning a Deep embedding model for zero-Shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, Hawaii, USA, 21-26 July 2017; pp. 3010–3019. [Google Scholar]
12. Zhu, X.; Zhang, R.; He, B. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, Vancouver, Canada, 18-22 June 2023; pp. 2639–2650. [Google Scholar]
13. Song, Y.; Liang, N.; Guo, Q. MeshCLIP: Efficient cross-modal information processing for 3D mesh data in zero/few-shot learning. Information processing and management 2023, 60, 103497. [Google Scholar] [CrossRef]
14. Kirk, R. A survey of zero-shot generalisation in deep reinforcement learning. Journal of artificial intelligence research 2023, 76, 201–264. [Google Scholar] [CrossRef]
15. Ball, L.; Parker, H. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. Proceedings of international conference on machine learning, Graz, Austria, 18-24 July 2021; pp. 619–629. [Google Scholar]
16. Chen, Y. UAV path planning using artificial potential field method updated by optimal control theory. International journal of systems science 2016, 47, 1407–1420. [Google Scholar] [CrossRef]
17. Botvinick; Michael, M. Michael. Hierarchical reinforcement learning and decision making. Current opinion in neurobiology. Current opinion in neurobiology 2012, 22, 956–962. [Google Scholar] [CrossRef] [PubMed]
18. Eppe, M.; Kerzel, C. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nature machine intelligence 2022, 4, 11–20. [Google Scholar] [CrossRef]
19. Wöhlke, J.; Schmitt, F.; Van, H. Hierarchies of planning and reinforcement learning for robot navigation. Proceedings of IEEE international conference on robotics and automation, Xi’an, China, May 30-June 6 2021; pp. 10682–10688. [Google Scholar]
20. Duan, J.; Eben, S.; Guan, Y. Hierarchical reinforcement learning for self-driving decision-making without reliance on labeled driving data. Intelligent transport systems 2020, 14, 297–305. [Google Scholar] [CrossRef]
21. Christen, S.; Jendele, L.; Aksan, E. Learning functionally decomposed hierarchies for continuous control tasks with path planning. IEEE robotics and automation letters 2021, 6, 3623–3630. [Google Scholar] [CrossRef]
22. Ye, X.; Yang, Y. Efficient robotic object search via hiem: Hierarchical policy learning with intrinsic-extrinsic modeling. IEEE robotics and automation letters 2021, 6, 4425–4432. [Google Scholar] [CrossRef]
23. Lin, B.; Zhu, Y.; Chen, Z. Adapt: Vision-language navigation with modality-aligned action prompts. In Proceedings of the IEEE conference on computer vision and pattern recognition, New Orleans, USA, 18-24 June 2022; pp. 15396–15406. [Google Scholar]
24. Kotei, E.; Thirunavukarasu, R. A systematic review of transformer-based pre-trained language models through self-supervised learning. Information 2023, 14, 187. [Google Scholar] [CrossRef]
25. Jaiswal, A..; Babu, A.; Zadeh, M. A survey on contrastive self-supervised learning. Technologies 2020, 9, 2. [Google Scholar] [CrossRef]
26. Huo, X.; Karimi, H.; Zhao, H. Adaptive-critic design for decentralized event-triggered control of constrained nonlinear interconnected systems within an identifier-critic framework. IEEE transactions on cybernetics 2021, 52, 7478–7491. [Google Scholar] [CrossRef] [PubMed]
27. Vaswani, A.; Shazeer, N.; Parmar, N. Attention is all you need. Proceedings of the conference on neural information processing systems 6000–6010.
28. Tiwari, R.; Killamsetty, K.; Iyer, R. Gcr: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, New Orleans, USA, 19-24 June 2022; pp. 99–108. [Google Scholar]
29. Marius, C.; Mohamed, O.; Sebastian, R. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, USA, 27-30 June 2016; pp. 3213–3223. [Google Scholar]