1. Introduction

2. Related work

3. System Design

3.3. Mask repair

The object mask segmented by Mask R-CNN usually has the problem of incompleteness, which may make part of the information of the dynamic object leak into the environment and affect the accuracy of pose estimation and map construct. To this end, we propose a new mask repair model to solve the incompleteness of the mask. The morphological method and the depth value fusion method are used to repair the mask for the small objects and the large objects, respectively.

3.3.1. Mask repair for small objects

When the object is relatively small, the corresponding area in the depth map often has an incomplete or hollow parts, so the depth map is not appropriate to improve the mask of small objects is not appropriate. Therefore, the morphological method is more suitable for improving the incompleteness of small object masks.

As shown in Figure 3(a), a small ball is slowly rolling on the table, and Figure 3(b) is the initial mask image of the RGB image after Mask R-CNN segmentation. It can be seen from the figure that although the segmented mask image is close to the outline of the small ball, its edge part is still quite different from the complete object outline. To improve the mask’s incompleteness, the median filter is first used to process the mask, which aims to eliminate the potential speckle noise in the small object mask and protect the edge part of the mask. Then, the erosion and dilation operation is used to make the edge part of the mask closer to the original object as much as possible. Finally, the improved small object mask image is shown in Figure 3(c). It can be seen from the figure that the inpainted mask is closer to the original object shape than the original mask.

3.3.2. Mask repair for large objects

The Morphological method cannot effectively solve the incompleteness of relatively large mask, and even complicate the incompleteness of the mask. Therefore the depth map is used to repair the mask.

A threshold $\tau$ is given, and the objects are treated as large objects when the ratio of the segmented object mask area to the entire RGB image area is greater than $\tau$. On the contrary, the objects are treated as small objects. The semantic information of the large object obtained by the segmentation network is used to obtain the object label, such as $Ob{j}_i$, where i ∈1,2,3...,n. Then each depth value of the depth map area $dept{h}_{mask}^{Ob{j}_i}$ corresponding to its mask area is added up to obtain the depth value set $dept{h}^{Ob{j}_i}$ of the object. Maximum, minimum, and average depth values in the depth set can be respectively represented as follows:

$$d_{min}=\mathrm{GetMin}\left(dept{h}^{Ob{j}_i}\right)$$

(1)

$$d_{max}=\mathrm{GetMax}\left(dept{h}^{Ob{j}_i}\right)$$

(2)

$$d_{avg}=\mathrm{GetAvg}\left(\frac{{\sum }_{i=1}^{n}dept{h}_i^{Ob{j}_i}}{n}\right)$$

(3)

here n represents the number of pixels, and $dept{h}_i^{Ob{j}_i}$ represents the depth value of ith pixel. To obtain the range of depth value fusion, $e_{max}$ and $e_{min}$ are first obtained by subtracting the average value $d_{avg}$ from the maximum $d_{max}$ and the minimum $d_{min}$, respectively, where $e_{min}$ takes the absolute value. Then, the range of depth value fusion can be obtained from formulas 6 and 7.

$$e_{max}=d_{max}-d_{avg}$$

(4)

$$e_{min}=\left|d_{min}-d_{avg}\right|$$

(5)

$${\tau }_{max}=\frac{e_{max}}{d_{avg}}{d}_{max}$$

(6)

$${\tau }_{min}=\frac{e_{min}}{d_{avg}}{d}_{min}$$

(7)

For more efficient fusion, a square slider with a size of 2*2 is used to traverse the depth image. When the slider traverses to the edge area, the mask depth map area is expanded by adding points whose depth value variation stays within the range of depth value of target objects to the mask depth area.

$$\left\{\begin{array}{cc}{d}_{min}-{\tau }_{min}\le d(u,v)\le d_{max}+{\tau }_{max}\hfill & (u,v)\in dept{h}_{mask}\hfill \\ else\hfill & (u,v)\notin dept{h}_{mask}\hfill \end{array}\right.$$

(8)

where $d_{u,v}$ represents the depth value on the $(u,v)$ pixel. Finally, the original mask is replaced by the fused depth region represented as a binary image. Algorithm 1 gives the specific steps of mask restoration. As shown in Figure 4, the optimized mask is closer to the contour of the object region than the original mask.

Figure 4. Comparison of mask repair for large objects.

Figure 4. Comparison of mask repair for large objects.

Algorithm 1 Mask improvement algorithm

Input: Current depth image $de{p}_c$, original Mask $Mas{k}_o$, mask size threshold $\tau$, rgb image size $Are{a}_{img}$ Output: revised mask 1: Label the objects $Ob{j}_1,...,Ob{j}_n$ in $Mas{k}_o$;   2: for each mask $Mas{k}_o^{Ob{j}_i}$ of $Ob{j}_i$ do   3:     Obtain a depth map $dept{h}_{mask}^{Ob{j}_i}$ of the corresponding area of the mask $Mas{k}_o^{Ob{j}_i}$   4:     Calculate the area $Ar{e}_{mask}^{Ob{j}_i}$ of $Mas{k}_o^{Ob{j}_i}$   5:     if $Ar{e}_{mask}^{Ob{j}_i}$≥$\tau$ then   6:         Record depth values $dept{h}^{Ob{j}_i}$ in $Mas{k}^{Ob{j}_i}$   7:         $d_{min}=\mathrm{GetMin}\left(dept{h}_n^{Ob{j}_i}\right)$;   8:         $d_{max}=\mathrm{GetMax}\left(dept{h}_n^{Ob{j}_i}\right)$;   9:         $d_{avg}=\mathrm{GetAvg}\left(\frac{{\sum }_{i=1}^{n}dept{h}_i^{Ob{j}_i}}{n}\right)$;   10:         $e_{max}=d_{max}-d_{avg}$,$e_{min}=\left|d_{min}-d_{avg}\right|$;   11:         ${\tau }_{max}=\frac{e_{max}}{d_{avg}}{d}_{max}$,${\tau }_{min}=\frac{e_{min}}{d_{avg}}{d}_{min}$;   12:         for each $point(u,v)$ in $dept{h}_{a}re{a}^i$ do   13:            if $d_{min}-{\tau }_{min}\le d(u,v)\le d_{max}+{\tau }_{max}$ then   14:                $(u,v)\in dept{h}_{mask}^i$   15:            else  16:                $(u,v)\notin dept{h}_{mask}^i$   17:            end if  18:         end for        Represent the fused depth region with a binary image   19:     else  20:           Repair $Mas{k}_o^{Ob{j}_i}$ with morphological methods   21:     end if  22: end for  23: return result

4. Experimental results

4.3. Ablation experiment

To verify the effectiveness of Mask R-CNN and mask repair on the SLAM system, this paper uses the original ORB-SLAM2 system without Mark R-CNN and mask repair as a baseline to complete the ablation experiments on the high dynamic sequence of the TUM RGBD dataset. The ATE is used for quantitative evaluation, and the quantitative and the qualitative results are given. The quantitative results are shown in the Table 5, and the qualitative results are shown in the Figure 9. From the results shown in Table 5 and Figure 9, we can find that the performance improvement in dynamic environments is obvious.

Table 5. Quantitative results of ablation experiments on the TUM RGBD dataset(ATE) [m].

Table 5. Quantitative results of ablation experiments on the TUM RGBD dataset(ATE) [m].

	fr3/w/half		fr3/w/rpy		fr3/w/xyz		fr3/w/static
	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.
Baseline	0.0453	0.2524	0.5391	0.2283	0.7521	0.4712	0.3194	0.1819
Baseline+Mask R-CNN	0.0260	0.0129	0.0295	0.0170	0.0175	0.0091	0.0062	0.0029
Baseline+Mask R-CNN+Mask repair	0.0257	0.0128	0.0288	0.0161	0.0166	0.0082	0.0060	0.0027

Figure 9. Qualitative results of ablation experiments on the TUM RGBD dataset. The first row is the result of the original ORB-SLAM2 experiment. The second row is the experimental result after adding the Mask-RCNN module to the original ORB-SLAM2. The experimental results of adding Mask-RCNN and mask repair module simultaneously on the original ORB-SLAM2 are shown in the third row.

Figure 9. Qualitative results of ablation experiments on the TUM RGBD dataset. The first row is the result of the original ORB-SLAM2 experiment. The second row is the experimental result after adding the Mask-RCNN module to the original ORB-SLAM2. The experimental results of adding Mask-RCNN and mask repair module simultaneously on the original ORB-SLAM2 are shown in the third row.

5. Conclusion

References

1. Durrant-Whyte, H.; Bailey, T. Simultaneous localization and mapping: part I. IEEE robotics & automation magazine 2006, 13, 99–110. [Google Scholar]
2. Yurtsever, E.; Lambert, J.; Carballo, A.; Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE access 2020, 8, 58443–58469. [Google Scholar] [CrossRef]
3. Huang, H.; Gartner, G. A survey of mobile indoor navigation systems. Cartography in Central and Eastern Europe 2009, 305–319. [Google Scholar]
4. Azuma, R.T. A survey of augmented reality. Presence: teleoperators & virtual environments 1997, 6, 355–385. [Google Scholar]
5. Mistry, I.; Tanwar, S.; Tyagi, S.; Kumar, N. Blockchain for 5G-enabled IoT for industrial automation: A systematic review, solutions, and challenges. Mechanical systems and signal processing 2020, 135, 106382. [Google Scholar] [CrossRef]
6. Mur-Artal, R.; Tardós, J.D. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics 2017, 33, 1255–1262. [Google Scholar] [CrossRef]
7. Engel, J.; Koltun, V.; Cremers, D. Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence 2017, 40, 611–625. [Google Scholar] [CrossRef]
8. Engel, J.; Schöps, T.; Cremers, D. LSD-SLAM: Large-scale direct monocular SLAM. In Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014; Proceedings, Part II 13. Springer: Berlin/Heidelberg, Germany, 2014; pp. 834–849. [Google Scholar]
9. Klein, G.; Murray, D. Parallel tracking and mapping for small AR workspaces. In Proceedings of the 2007 6th IEEE and ACM international symposium on mixed and augmented reality; IEEE: Piscataway, NJ, USA, 2007; pp. 225–234. [Google Scholar]
10. LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
11. Zhou, K.Q.; Gui, W.H.; Mo, L.P.; Zain, A.M. A bidirectional diagnosis algorithm of fuzzy Petri net using inner-reasoning-path. Symmetry 2018, 10, 192. [Google Scholar] [CrossRef]
12. Chen, C.F.; Zain, A.M.; Zhou, K.Q. Definition, approaches, and analysis of code duplication detection (2006–2020): a critical review. Neural Computing and Applications 2022, 1–31. [Google Scholar] [CrossRef]
13. Chen, Z.; Francis, A.; Li, S.; Liao, B.; Xiao, D.; Ha, T.T.; Li, J.; Ding, L.; Cao, X. Egret Swarm Optimization Algorithm: An Evolutionary Computation Approach for Model Free Optimization. Biomimetics 2022, 7, 144. [Google Scholar] [CrossRef]
14. Zhang, X.Y.; Zhou, K.Q.; Li, P.C.; Xiang, Y.H.; Zain, A.M.; Sarkheyli-Hägele, A. An Improved Chaos Sparrow Search Optimization Algorithm Using Adaptive Weight Modification and Hybrid Strategies. IEEE Access 2022, 10, 96159–96179. [Google Scholar] [CrossRef]
15. Xiao, L.; Liao, B.; Li, S.; Zhang, Z.; Ding, L.; Jin, L. Design and analysis of FTZNN applied to the real-time solution of a nonstationary Lyapunov equation and tracking control of a wheeled mobile manipulator. IEEE Transactions on Industrial Informatics 2017, 14, 98–105. [Google Scholar] [CrossRef]
16. Zhang, Y.; Qiu, B.; Liao, B.; Yang, Z. Control of pendulum tracking (including swinging up) of IPC system using zeroing-gradient method. Nonlinear Dynamics 2017, 89, 1–25. [Google Scholar] [CrossRef]
17. Xiao, L.; Zhang, Z.; Li, S. Solving time-varying system of nonlinear equations by finite-time recurrent neural networks with application to motion tracking of robot manipulators. IEEE Transactions on Systems, Man, and Cybernetics: Systems 2018, 49, 2210–2220. [Google Scholar] [CrossRef]
18. Li, J.; Peng, H.; Hu, H.; Luo, Z.; Tang, C. Multimodal information fusion for automatic aesthetics evaluation of robotic dance poses. International Journal of Social Robotics 2020, 12, 5–20. [Google Scholar] [CrossRef]
19. Peng, H.; Hu, H.; Chao, F.; Zhou, C.; Li, J. Autonomous robotic choreography creation via semi-interactive evolutionary computation. International Journal of Social Robotics 2016, 8, 649–661. [Google Scholar] [CrossRef]
20. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision; 2017; pp. 2961–2969. [Google Scholar]
21. Sturm, J.; Engelhard, N.; Endres, F.; Burgard, W.; Cremers, D. A benchmark for the evaluation of RGB-D SLAM systems. In Proceedings of the 2012 IEEE/RSJ international conference on intelligent robots and systems; IEEE: Piscataway, NJ, USA, 2012; pp. 573–580. [Google Scholar]
22. Saputra, M.R.U.; Markham, A.; Trigoni, N. Visual SLAM and structure from motion in dynamic environments: A survey. ACM Computing Surveys (CSUR) 2018, 51, 1–36. [Google Scholar] [CrossRef]
23. Kundu, A.; Krishna, K.M.; Sivaswamy, J. Moving object detection by multi-view geometric techniques from a single camera mounted robot. In Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems; IEEE: Piscataway, NJ, USA, 2009; pp. 4306–4312. [Google Scholar]
24. Zou, D.; Tan, P. Coslam: Collaborative visual slam in dynamic environments. IEEE transactions on pattern analysis and machine intelligence 2012, 35, 354–366. [Google Scholar] [CrossRef]
25. Sun, Y.; Liu, M.; Meng, M.Q.H. Improving RGB-D SLAM in dynamic environments: A motion removal approach. Robotics and Autonomous Systems 2017, 89, 110–122. [Google Scholar] [CrossRef]
26. Newcombe, R.A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A.J.; Kohi, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. Kinectfusion: Real-time dense surface mapping and tracking. In Proceedings of the 2011 10th IEEE international symposium on mixed and augmented reality; IEEE: Piscataway, NJ, USA, 2011; pp. 127–136. [Google Scholar]
27. Horn, B.K.; Schunck, B.G. Determining optical flow. Artificial intelligence 1981, 17, 185–203. [Google Scholar] [CrossRef]
28. Derome, M.; Plyer, A.; Sanfourche, M.; Besnerais, G.L. Moving object detection in real-time using stereo from a mobile platform. Unmanned Systems 2015, 3, 253–266. [Google Scholar] [CrossRef]
29. Derome, M.; Plyer, A.; Sanfourche, M.; Le Besnerais, G. Real-time mobile object detection using stereo. In Proceedings of the 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV); IEEE: Piscataway, NJ, USA, 2014; pp. 1021–1026. [Google Scholar]
30. Fang, Y.; Dai, B. An improved moving target detecting and tracking based on optical flow technique and kalman filter. In Proceedings of the 2009 4th International Conference on Computer Science & Education; IEEE: Piscataway, NJ, USA, 2009; pp. 1197–1202. [Google Scholar]
31. Wang, Y.; Huang, S. Towards dense moving object segmentation based robust dense RGB-D SLAM in dynamic scenarios. In Proceedings of the 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV); IEEE: Piscataway, NJ, USA, 2014; pp. 1841–1846. [Google Scholar]
32. Long, F.; Ding, L.; Li, J. DGFlow-SLAM: A Novel Dynamic Environment RGB-D SLAM without Prior Semantic Knowledge Based on Grid Segmentation of Scene Flow. Biomimetics 2022, 7, 163. [Google Scholar] [CrossRef] [PubMed]
33. Alcantarilla, P.F.; Yebes, J.J.; Almazán, J.; Bergasa, L.M. On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation; IEEE: Piscataway, NJ, USA, 2012; pp. 1290–1297. [Google Scholar]
34. Zhang, T.; Zhang, H.; Li, Y.; Nakamura, Y.; Zhang, L. Flowfusion: Dynamic dense rgb-d slam based on optical flow. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA); IEEE: Piscataway, NJ, USA, 2020; pp. 7322–7328. [Google Scholar]
35. Sun, D.; Yang, X.; Liu, M.Y.; Kautz, J. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2018; pp. 8934–8943. [Google Scholar]
36. Badrinarayanan, V.; Kendall, A.; Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 2017, 39, 2481–2495. [Google Scholar] [CrossRef] [PubMed]
37. Yu, C.; Liu, Z.; Liu, X.J.; Xie, F.; Yang, Y.; Wei, Q.; Fei, Q. DS-SLAM: A semantic visual SLAM towards dynamic environments. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE: Piscataway, NJ, USA, 2018; pp. 1168–1174. [Google Scholar]
38. Bescos, B.; Fácil, J.M.; Civera, J.; Neira, J. DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters 2018, 3, 4076–4083. [Google Scholar] [CrossRef]
39. Xiao, L.; Wang, J.; Qiu, X.; Rong, Z.; Zou, X. Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomous Systems 2019, 117, 1–16. [Google Scholar] [CrossRef]
40. Liu, Y.; Miura, J. RDS-SLAM: Real-time dynamic SLAM using semantic segmentation methods. IEEE Access 2021, 9, 23772–23785. [Google Scholar] [CrossRef]
41. Campos, C.; Elvira, R.; Rodríguez, J.J.G.; Montiel, J.M.; Tardós, J.D. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics 2021, 37, 1874–1890. [Google Scholar] [CrossRef]
42. Liu, Y.; Miura, J. RDMO-SLAM: Real-time visual SLAM for dynamic environments using semantic label prediction with optical flow. IEEE Access 2021, 9, 106981–106997. [Google Scholar] [CrossRef]
43. Xu, Y.; Wang, Y.; Huang, J.; Qin, H. ESD-SLAM: An efficient semantic visual SLAM towards dynamic environments. Journal of Intelligent & Fuzzy Systems 2022, 1–10. [Google Scholar]
44. Xie, W.; Liu, P.X.; Zheng, M. Moving object segmentation and detection for robust RGBD-SLAM in dynamic environments. IEEE Transactions on Instrumentation and Measurement 2020, 70, 1–8. [Google Scholar] [CrossRef]
45. Lucas, B.D.; Kanade, T. An iterative image registration technique with an application to stereo vision. In Proceedings of the IJCAI’81: 7th international joint conference on Artificial intelligence; 1981; Volume 2, pp. 674–679. [Google Scholar]