1. Introduction

Figure 1. Comparison of the motion aspects between human eye and electronic camera (photo courtesy of free source in Internet).

Figure 1. Comparison of the motion aspects between human eye and electronic camera (photo courtesy of free source in Internet).

2. Problem Statement

Figure 2. Outer loop of perception, planning and control inside autonomous robot arm manipulator and autonomous humanoid robot.

Figure 2. Outer loop of perception, planning and control inside autonomous robot arm manipulator and autonomous humanoid robot.

3. Related Works

3.2. Forward Projection Matrix of Camera

A single camera is the basis of a monocular vision. Before we could understand the 2D forward and inverse projections of monocular vision, it is necessary for us to know the details of a camera’s forward projection matrix.

Refer to Figure 4. With the use of the terminology of kinematic chain, the derivation of camera matrix starts with the transformation from reference frame to camera frame. If the coordinates of point Q with respect to reference frame are $(X,Y,Z)$, the coordinates $(X_c,Y_c,Z_c)$ of the same point Q with respect to camera frame will be [12]:

$$\left[\begin{array}{c}{X}_c\\ Y_c\\ \begin{array}{c}{Z}_c\\ 1\end{array}\end{array}\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& \begin{array}{cc}{r}_{13}& t_x\end{array}\\ r_{21}& r_{22}& \begin{array}{cc}{r}_{23}& t_y\end{array}\\ \begin{array}{c}{r}_{31}\\ 0\end{array}& \begin{array}{c}{r}_{32}\\ 0\end{array}& \begin{array}{cc}\begin{array}{c}{r}_{33}\\ 0\end{array}& \begin{array}{c}{t}_z\\ 1\end{array}\end{array}\end{array}\right]\bullet \left[\begin{array}{c}X\\ Y\\ \begin{array}{c}Z\\ 1\end{array}\end{array}\right]$$

(1)

where rotation matrix $\{r_{ij},i\in \left[\mathrm{1,3}\right],j\in [\mathrm{1,3}\left]\right\}$ represents the orientation of reference frame with respect to camera frame, and translation vector ${(t_x,t_y,t_z)}^t$ represents the position of reference frame’s origin with respect to camera frame.

Inside the camera frame, the transformation from the coordinates $(X_c,Y_c,Z_c)$ of point Q to the analogue image coordinates ${(x,y)}^t$ of point q will be:

$$\left[\begin{array}{c}s\bullet x\\ s\bullet y\\ s\end{array}\right]=\left[\begin{array}{ccc}f& 0& \begin{array}{cc}0& 0\end{array}\\ 0& f& \begin{array}{cc}0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\bullet \left[\begin{array}{c}{X}_c\\ Y_c\\ \begin{array}{c}{Z}_c\\ 1\end{array}\end{array}\right]$$

(2)

where $f$ is the focal length of the camera and $s$ is a scaling factor.

By default, we are using digital cameras. Hence, an analogue image is converted into its corresponding digital image. Such process of digitization results in the further transformation from analogue image frame to digital image frame. This transformation is described by the following equation:

$$\left[\begin{array}{c}s\bullet u\\ s\bullet v\\ s\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{∆u}& 0& u_0\\ 0& \frac{1}{∆v}& v_0\\ 0& 0& 1\end{array}\right]\bullet \left[\begin{array}{c}s\bullet x\\ s\bullet y\\ s\end{array}\right]$$

(3)

where ${(u,v)}^t$ are the digital image coordinates of point q, $∆u$ is the width of a pixel (i.e., a digital image’s pixel density in horizontal direction), $∆v$ is the height of a pixel (i.e., a digital image’s pixel density in vertical direction), and ${(u_0,v_0)}^t$ are the digital image coordinates of the intersection point between the optical axis (i.e., camera frame’s Z axis) and the image plane (note: this point is also called a camera’s principal point).

Now, by substituting Equations (1) and (2) into Equation (3), we will be able to obtain the following equation [16]:

$$\left[\begin{array}{c}s\bullet u\\ s\bullet v\\ s\end{array}\right]=C_f\bullet \left[\begin{array}{c}X\\ Y\\ \begin{array}{c}Z\\ 1\end{array}\end{array}\right]$$

(4)

with

$$C_f=\left[\begin{array}{ccc}\frac{f}{∆u}& 0& \begin{array}{cc}{u}_0& 0\end{array}\\ 0& \frac{f}{∆v}& \begin{array}{cc}{v}_0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\bullet \left[\begin{array}{ccc}{r}_{11}& r_{12}& \begin{array}{cc}{r}_{13}& t_x\end{array}\\ r_{21}& r_{22}& \begin{array}{cc}{r}_{23}& t_y\end{array}\\ \begin{array}{c}{r}_{31}\\ 0\end{array}& \begin{array}{c}{r}_{32}\\ 0\end{array}& \begin{array}{cc}\begin{array}{c}{r}_{33}\\ 0\end{array}& \begin{array}{c}{t}_z\\ 1\end{array}\end{array}\end{array}\right]$$

(5)

where matrix $C_f$ is called a camera’s forward projection matrix which is a $3\times 4$ matrix.

Figure 4. A single camera is the basis of a monocular vision.

Figure 4. A single camera is the basis of a monocular vision.

3.3. 3D Forward Projection of Monocular Vision

A monocular vision system uses a single camera. Its kinematic chain is the same as the one shown in Figure 4. Most importantly, Equation (4) describes 3D forward projection of a monocular vision system, in which 3D coordinates ${(X,Y,Z)}^t$ are projected into 2D digital image coordinates ${(u,v)}^t$.

3.4. 3D Inverse Projection of Monocular Vision

From the viewpoint of pure mathematics, Equation (4) could be re-written into the following form:

$$\left[\begin{array}{c}k\bullet X\\ k\bullet Y\\ \begin{array}{c}k\bullet Z\\ k\end{array}\end{array}\right]=C_i\bullet \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(6)

with

$${C_i=(C_f^t\bullet C_f)}^{-1}\bullet C_f^t$$

(7)

and $k=1/s$. It is worth noting that Equation (6) has never been explicitly described in any textbook because it is useless in theory and in practice. However, Equation (6) has inspired us to derive Equation (20) which represents an important advance in computer vision, robot vision, and artificial intelligence.

In theory, Equation (6) describes 3D inverse projection of a monocular vision system. In practice, Equation (6) could be graphically represented by an artificial neural network which serves as predictor. The input layer consists of ${(u,v,1)}^t$ and the output layer consists of ${(X,Y,Z)}^t$. Matrix $C_i$ contains the weighting coefficients. Hence, it is clear to us that a different matrix $C_i$ will enable the prediction of coordinates ${(X,Y,Z)}^t$ on a different planar surface. Most importantly, matrix $C_i$ could be obtained by a top-down process of calibration or a bottom-up process of tuning (i.e., optimization). Therefore, Equation (6) serves as a good example which helps us to understand the difference between machine learning and machine calibration (or tuning).

Although $C_i$ is a $4\times 3$ matrix, it is not possible to use Equation (6) to generally compute 3D coordinates ${(X,Y,Z)}^t$ in an analogue scene from 2D index coordinates ${(u,v)}^t$ (i.e., u is column index while v is row index) in a digital image. However, the philosophy behind Equation (6) has inspired us to discover a similar, but very useful, 3D inverse projection of binocular vision which will be described in Section 4.

3.5. 2D Forward Projection of Monocular Vision

Refer to Figure 4. If we consider the points or locations on the OXY plane of reference frame, Z coordinate in Equation (4) becomes zero. Hence, Equation (4) could be re-written into the following form:

$$\left[\begin{array}{c}s\bullet u\\ s\bullet v\\ s\end{array}\right]=M_f\bullet \left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]$$

(8)

where matrix $M_f$ is the version of matrix $C_f$ after removing its third column because Z is equal to zero. Clearly, matrix $M_f$ is a $2\times 2$ matrix and is invertible. As shown in Figure 8, Equation (8) describes the 2D forward projection from coordinates ${(X,Y)}^t$ on a plane of reference frame into digital image coordinates ${(u,v)}^t$ of monocular vision.

3.6. 2D Inverse Projection of Monocular Vision

Now, by inverting Equation (8), we could easily obtain the following result:

$$\left[\begin{array}{c}k\bullet X\\ k\bullet Y\\ k\end{array}\right]=M_i\bullet \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(9)

with

$$M_i={(M_f^t\bullet M_f)}^{-1}\bullet M_f^t$$

(10)

where matrix $M_i$ is also a $2\times 2$ matrix.

It goes without saying that Equations (8) and (9) fully describe 2D forward and inverse projections of a monocular vision system as shown in Figure 5.

4. Equations of 3D Projection in Human-like Binocular Vision

4.4. Equation of 3D Forward Projection of Displacement of Binocular Vision

Now, by doing a simple pseudo-inverse of matrix $D_i$, Equation (22) will allow us to obtain the following equation of 3D forward projection of displacement in binocular vision:

$$\left[\begin{array}{c}∆u_l\\ ∆v_l\\ \begin{array}{c}∆u_r\\ ∆v_r\end{array}\end{array}\right]=D_f\bullet \left[\begin{array}{c}∆X\\ ∆Y\\ ∆Z\end{array}\right]$$

(23)

where $D_f={(D_i^t\bullet D_i)}^{-1}\bullet D_i^t$.

Hence, Equations (22) and (23) fully describe 3D forward and inverse projections of displacement in a binocular vision system. These two solutions are iterative in nature and could be used inside the outer loop of perception, planning and control of autonomous robots as shown in Figure 7.

Especially, Equation (22) enables autonomous robots to achieve human-like hand-eye coordination and leg-eye coordination as shown in Figure 7. For example, a control task of hand-eye coordination or leg-eye coordination could be defined as the goal which is to minimize error vector ${(∆u_l,∆v_l,∆u_r,∆v_r)}^t$. As illustrated in Figure 7, the history of error vector ${(∆u_l,∆v_l,∆u_r,∆v_r)}^t$ will appear as paths which could be observed inside both left and right images.

Figure 7. Scenarios of achieving human-like hand-eye coordination and leg-eye coordination.

Figure 7. Scenarios of achieving human-like hand-eye coordination and leg-eye coordination.

5. Experimental Results

5.1. Real Experiment Validating Equation of 3D Inverse Projection of Position

Here, we would like to share an experiment in which we makes use of low-cost hardware with low-resolution binocular cameras and a small-sized checkerboard. In this way, we could appreciate the validity of Equation (20) and the theoretical result which is summarized in Figure 6.

As shown in Figure 8, the experimental hardware includes a Raspberry Pi single board computer, a binocular vision module, and a checkerboard. The image resolution of the binocular cameras is $480\times 320$ pixels. The checkerboard has the size of $18\times 24$ cm, which is divided into $6\times 8$ squares with the size of $3.0\times 3.0$ cm each.

Figure 8. Experimental hardware includes Raspberry Pi single board computer with a binocular vision module and a checkerboard which serves as input of calibration data-points as well as test data-points.

Figure 8. Experimental hardware includes Raspberry Pi single board computer with a binocular vision module and a checkerboard which serves as input of calibration data-points as well as test data-points.

Inside the checkerboard, $\{A,B,C,D\}$ serve as calibration data-points for the purpose of determining matrix $B_i$ in Equation (20), while $\{T_0,T_1,T_2,T_3,T_4\}$ serve as test data-points of the calibration result (i.e., to test the validity of matrix $B_i$ in Equation (20)).

Refer to Equation (20), matrix $B_i$ is a $4\times 5$ matrix in which there are nineteen independent elements or parameters. Since a single Equation (20) will impose three constraints, at least seven pairs of $\{X,Y,Z\}$ and $\{u_l,v_l,u_r,v_r\}$ are needed for us to fully determine matrix $B_i$.

As shown in Figure 9, we define a reference coordinate system as follows: Its Z axis is parallel to the ground and is pointing toward the scene. Its Y axis is perpendicular to the ground and is pointing downward. Its X axis is pointing toward the right-hand side.

Then, we place the checkerboard at four locations in front of the binocular vision system. The Z coordinates of these four locations are 1.0 m, 1.5 m, 2.0 m, and 2.5 m, respectively. The checkerboard is perpendicular to Z axis, which passes through test data-point $T_0$. Therefore, the X and Y coordinates of the calibration data-points $\{A,B,C,D\}$ and the test data-points $\{T_0,T_1,T_2,T_3,T_4\}$ are known in advance. The values of these X and Y coordinates are shown inside Figure 9.

Figure 9. Data set for calibrating matrix $B_i$ in Equation (20).

Figure 9. Data set for calibrating matrix $B_i$ in Equation (20).

When the checkerboard is placed at one of the above-mentioned four locations, a pair of stereo images is taken. The index coordinates of the calibration data-points and the test data-points could be determined either automatically or manually.

By putting the 3D coordinates and index coordinates of the calibration data-points together, we obtain Table 1 which contains the data needed for calibrating the equation of 3D inverse projection of binocular vision (i.e., Equation (20)).

With the use of data listed in Table 1, we obtain the following result of matrix $B_i$:

$$B_i=\left[\begin{array}{ccc}-0.4251& -0.7861& \begin{array}{ccc}-0.2245& 0.8267& 92.6220\end{array}\\ 0.2167& -0.3717& \begin{array}{ccc}-0.2730& 0.9845& -196.0451\end{array}\\ \begin{array}{c}-1.5409\\ -0.0874\end{array}& \begin{array}{c}14.4961\\ 0.1758\end{array}& \begin{array}{ccc}\begin{array}{c}1.2774\\ 0.0873\end{array}& \begin{array}{c}-14.9300\\ -0.1758\end{array}& \begin{array}{c}-71.2214\\ 1.0000\end{array}\end{array}\end{array}\right]$$

(24)

Now, we use the index coordinates in Table 1, calibrated matrix $B_i$, and Equation (20) to calculate the 3D coordinates of calibration data-points $\{A,B,C,D\}$. By combining these calculated 3D coordinates with data in Table 1, we will obtain Table 2 which helps us to compare between the true values of $\left\{A,B,C,D\right\}$’s 3D coordinates and the calculated values of $\{A,B,C,D\}$’s 3D coordinates.

In Table 2, the values in columns 2, 5 and 8 are the ground truth data of (X, Y, Z) coordinates of $\left\{A,B,C,D\right\}$. The values in columns 3, 6, and 9 are the computed values of $\left\{A,B,C,D\right\}$‘s (X, Y, Z) coordinates by using Equation (20).

Similarly, we use the index coordinates of the test data-points $\{T_0,T_1,T_2,T_3,T_4\}$, calibrated matrix $B_i$, and Equation (20) to calculate the 3D coordinates of $\{T_0,T_1,T_2,T_3,T_4\}$. Then, by combining the true values of $\{T_0,T_1,T_2,T_3,T_4\}$’s 3D coordinates and the calculated values of $\{T_0,T_1,T_2,T_3,T_4\}$’s 3D coordinates together, we obtain Table 3 which helps us to appreciate the usefulness and validity of Equation (20).

In Table 3, the values in columns 2, 5 and 8 are the ground truth data of (X, Y, Z) coordinates of ${\{T}_0,T_1,T_2,T_3,T_4\}$. The values in columns 3, 6, and 9 are the computed values of ${\{T}_0,T_1,T_2,T_3,T_4\}$‘s (X, Y, Z) coordinates by using Equation (20).

In view of the low-resolution of digital images (i.e., $480\times 320$ pixels) and a small-sized checkerboard (i.e., $18\times 24$ cm divided into $6\times 8$ squares), we could say that the comparison results shown in Table 2 and Table 3 are reasonably good enough for us to experimentally validate Equation (20). In practice, images with much higher resolutions and checkerboards of larger sizes will naturally increase the accuracy of binocular vision calibration as well as the accuracy of calculated 3D coordinates by using Equation (20).

6. Conclusion

References

1. Xie, M. Hu, Z. C. and Chen, H. New Foundation of Artificial Intelligence. World Scientific, 2021.
2. Horn, B. K. P. Robot Vision. The MIT Press, 1986.
3. Tolhurst, D. J. Sustained and transient channels in human vision. Vision Research, 1975; Volume 15, Issue 10, pp1151-1155.
4. Fahle M and Poggio, T. Visual hyperacuity: spatiotemporal interpolation in human vision, Proceedings of Royal Society, London, 1981.
5. Enns, J. T. and Lleras, A. What’s next? New evidence for prediction in human vision. Trends in Cognitive Science 2008, 12, 327–333. [Google Scholar] [CrossRef] [PubMed]
6. Laha, B. , Stafford, B. K. and Huberman, A. D. Regenerating optic pathways from the eye to the brain. Science, 2017, 356, 1031–1034. [Google Scholar] [CrossRef] [PubMed]
7. Gregory, R. Eye and Brain: The Psychology of Seeing - Fifth Edition, The Princeton University Press, 2015.
8. Pugh, A. (editor). Robot Vision. Springer-Verlag, 2013.
9. Samani, H. (editor). Cognitive Robotics. CRC Press, 2015.
10. Erlhagen, W. and Bicho, E. The dynamic neural field approach to cognitive robotics. Journal of Neural Engineering 2006, 3.
11. Cangelosi, A. and Asada, M. Cognitive Robotics. The MIT Press, 2022.
12. Faugeras, O. Three-dimensional Computer Vision: A Geometric Viewpoint. The MIT Press, 1993.
13. Paragios, N. , Chen, Y. M. and Faugeras, O. (editors). Handbook of Mathematical Models in Computer Vision. Springer, 2006.
14. Faugeras, O. Luong, Q. T. and Maybank, S. J. Camera self-calibration: Theory and experiments. European Conference on Computer Vision. Springer, 1992; LNCS, Volume 588.
15. Stockman, G. and Shapiro, L. G. Computer Vision. Prentice Hall, 2001.
16. Shirai, Y. Three-Dimensional Computer Vision. Springer, 2012.
17. Khan, S. Rahmani, H., Shah, S. A. A. and Bennamoun, M. A Guide to Convolutional Neural Networks for Computer Vision. Springer, 2018.
18. Szeliski, R. Computer Vision: Algorithms and Applications. Springer, 2022.
19. Brooks, R. New Approaches to Robotics. Science 1991, 253. [Google Scholar] [CrossRef] [PubMed]
20. Xie, M. Fundamentals of Robotics: Linking Perception to Action. World Scientific, 2003.
21. Siciliano, B. and Khatib, O. Springer Handbook of Robotics. Springer, 2016.
22. Murphy, R. Introduction to AI Robotics - Second Edition. The MIT Press, 2019.
23. Clarke, T. A. and Fryer, J. G. (1998). The development of camera calibration methods and models. The Photogrammetric Record, Wiley Online Library.
24. Zhang, Z. Y. , A flexible new technique for camera calibration. IEEE Transactions on Pattern Recognition and Machine Intelligence 2000, 22, 1330–1334. [Google Scholar] [CrossRef]
25. Wang, Q,, Fu, L. and Liu Z. Z (2010). Review on camera calibration, 2010 Chinese Control and Decision Conference, Xuzhou, 2010, pp. 3354-3358. [CrossRef]
26. Cui, Y., Zhou, F. Q., Wang, Y. X., Liu, L., and Gao, H. Precise calibration of binocular vision system used for vision measurement, Optics Express 2014, 22, 9134-9149.
27. Zhang, Y. J. (2023). Camera Calibration. In: 3-D Computer Vision. Springer, Singapore.
28. Fetić, A. Jurić, D. and Osmanković, D. (2012). The procedure of a camera calibration using Camera Calibration Toolbox for MATLAB, 2012 Proceedings of the 35th International Convention MIPRO, Opatija, Croatia, pp. 1752-1757.
29. Xu, G., Chen. J. and Li, X. 3-D Reconstruction of Binocular Vision Using Distance Objective Generated From Two Pairs of Skew Projection Lines, IEEE Access 2017, 5, 27272–27280. [Google Scholar] [CrossRef]
30. Xu, B. Q. and Liu, C. A 3D reconstruction method for buildings based on monocular vision. Computer-aided Civil and Infrastructure Engineering 2021, 37, 354–369. [Google Scholar] [CrossRef]