1. Introduction

2. Literature Review and Related Work

3. Problem Formulation

3.2. Formulation of Discrete Optimal Transport Problem

We suppose the 2 dimensional image of the detecting image and the reference image are both in the size of $d_1\times d_2$ pixels for some $d_1,d_2\in \mathbb{N}$, say

$$X,Y\subseteq \mathbb{Z}\times \mathbb{Z}$$

respectively. Since $X,Y$ are in same size, we can easily transform X to Y by elementary arithmetic, thus we assume $X=Y$. Similarly, we assume

$$X=Y=\{(x,y):x\in [0,d_1-1],y\in [0,d_2-1]\}$$

Consider the gray scale

$$S_{gray}=\{0,1,2,...,\kappa \}$$

for some $\kappa \in \mathbb{N}$ represents the brightness of the image, where 0 and $\kappa$ represent the darkest and brightest respectively.

We consider $f^{*},g^{*}$ to be densities on $X,Y$ such that for all $x\in X,y\in Y$, we have $f^{*}\left(x\right),g^{*}\left(y\right)\in S_{gray}$. By normalizing $f^{*},g^{*}$, we have the normalized image (density/measure) of the detecting image f and reference image g where

$$\begin{array}{ccc}\hfill f\left(x\right)={\displaystyle \frac{f^{*}\left(x\right)}{{\sum }_{\alpha \in X}{f}^{*}\left(\alpha \right)}}& \hfill & \hfill g\left(y\right)={\displaystyle \frac{g^{*}\left(y\right)}{{\sum }_{\beta \in Y}{g}^{*}\left(\beta \right)}}\end{array}$$

Then the optimal transport problem is as follows:

$$\begin{array}{ccc}\hfill C_{optimal}=min\left\{{\int }_{X\times Y}c(x,y)d\gamma (x,y)\right|\gamma \in \Pi (f,g)\}& \hfill & \hfill c={\parallel x-y\parallel }_2\end{array}$$

(6)

where the cost function c is the Euclidean distance. If $C_{optimal}$ exist, it is the optimal transport cost and the map $\gamma$ is the optimal transport map.

Table 1. Notations in Problem Formulation.

Table 1. Notations in Problem Formulation.

Notation	Description
$d_1\times d_2$	The size in pixel of the input image
X	The 2 dimensional coordinates for each pixel in detecting image
Y	The 2 dimensional coordinates for each pixel in reference image
$S_{gray}$	The set of gray scale
$\kappa$	Maximum of gray scale
$f^{*}$	The measure on X representing the gray sacle detecting image
$g^{*}$	The measure on Y representing the gray sacle reference image
f	The normalized measure of $f^{*}$ on X
g	The normalized measure of $g^{*}$ on Y
$C_{optimal}$	The optimal cost of transporting f to g

4. Solver Design

4.1. 2 Dimensional to 1 Dimensional

For further simplicity, we could transform the assignment problem in Section 3 to 1 dimensional form. We first consider a transform function $\mathcal{T}:X\to \mathbb{N}$ such that

$$\begin{array}{c}\hfill \mathcal{T}(x,y)=x+d_{2}y\end{array}$$

(7)

Clearly, $\mathcal{T}$ is bijective. Denote $k=d_1{d}_2-1$, and we let $\mathcal{D}=\mathcal{T}\left(X\right)=[0,k]\in \mathbb{N}$ to be the 1 dimensional frame for images of detecting image and reference image, and we let $\widehat{f},\widehat{g}$ to be the 1 dimensional representation of the images where $\widehat{f}\left(x\right)=f\left({\mathcal{T}}^{-1}\left(x\right)\right)$ and $\widehat{g}\left(y\right)=g\left({\mathcal{T}}^{-1}\left(y\right)\right)$. Then the assignment problem becomes

$$\begin{array}{ccccc}\hfill min\sum _{i,j\in \mathcal{D}}{x}_{i,j}{c}_{i,j}& \hfill & \hfill s.t.& \hfill & \hfill \left\{\begin{array}{cc}\sum _{j\in \mathcal{D}}{x}_{i,j}=\widehat{f}\left(i\right),\hfill & \forall i\in \mathcal{D}\hfill \\ \sum _{i\in \mathcal{D}}{x}_{i,j}=\widehat{g}\left(j\right),\hfill & \forall j\in \mathcal{D}\hfill \end{array}\right.\end{array}$$

(8)

where $x_{i,j}$ represents the mass transports from ${\mathcal{T}}^{-1}\left(i\right)$ to ${\mathcal{T}}^{-1}\left(j\right)$

$$\begin{array}{c}\hfill c_{i,j}={\parallel {\mathcal{T}}^{-1}\left(i\right)-{\mathcal{T}}^{-1}\left(j\right)\parallel }_2\end{array}$$

(9)

Table 2. Notations in Encoding and Decoding.

Table 2. Notations in Encoding and Decoding.

Notation	Description
$\mathcal{T}$	The bijective (2D to 1D) flatten function
k	The number of pixels minus one
$\mathcal{D}$	The 1 dimensional coordinates for the gray scale of both input images
$\widehat{f}$	The measure on $\mathcal{D}$ representing the gray scale detecting image
$\widehat{g}$	The measure on $\mathcal{D}$ representing the gray scale reference image
$c_{i,j}$	The unit cost for transporting from ith coordinate to jth coordinate in $\mathcal{D}$
$x_{i,j}$	The mass to transport from ith coordinate to jth coordinate in $\mathcal{D}$
${\sum }_{i,j\in \mathcal{D}}$	The cost of a transport
${\sum }_{j\in \mathcal{D}}{x}_{i,j}$	Mass conservation constraints for detecting image at ith coordinate in $\mathcal{D}$
$=\widehat{f}\left(i\right)$
${\sum }_{i\in \mathcal{D}}{x}_{i,j}$	Mass conservation constraints for reference image at jth coordinate in $\mathcal{D}$
$=\widehat{g}\left(j\right)$
$\mathcal{C}$	The cost map
$\mathcal{X}$	The transport plan
$\mathcal{A}$	The constraint matrix which project $\mathcal{X}$ to vector in ${\left[\begin{array}{ccc}\mathcal{D}& & \mathcal{D}\end{array}\right]}^T$
$\mathcal{B}$	The constraint vector represents the gray scale at each point for both image

5. Discrete Optimal Transport Based Welding Defect Detection Method

5.2. Image to Matrix

We assume that the input image is in same size as it is assumed in Section 3.2. We also assume that the input image can be formulated as pixels of some ratio (0-255) of red, green, and blue (RGB). Therefore, an image in RGB can be considered as a matrix where each element is a vector in ${\mathbb{N}}^3$. For example:

5.3. Find Welding Area

Since the components on PCB have distinct color in comparison to the color of PCB, we consider matrix $R,G,B$ having same size as the RGB matrix such that $R,G,B$ contain only R,G,B value in RGB matrix respectively. We define

$$\begin{array}{ccccc}\hfill \mathcal{N}(M,i,j,k)={\displaystyle \frac{1}{{(2k+1)}^2}}\sum _{q\in [-k,k]}{m}_{i+q,j+q}& \hfill & \hfill s.t.& \hfill & \hfill \left\{\begin{array}{c}i\pm k\in [0,m]\hfill \\ j\pm k\in [0,n]\hfill \end{array}\right.\end{array}$$

(14)

for $i,j,k\in \mathbb{N},M\in {\mathbb{R}}^{m\times n}$, which represents the average value for elements near the element at ith row jth column of matrix M. Then for the $R,G,B$, say they all have size $m\times n$, we consider $c=(c_1,c_2)=(\lfloor {\displaystyle \frac{m}{2}}\rfloor ,\lfloor {\displaystyle \frac{n}{2}}\rfloor )$ represents the relative center element of the matrix. Let

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{l}_1=\{(0,c_2),(1,c_2),...,(m,c_2)\}\hfill \\ l_2=\{(c_1,0),(c_1,1),...,(c_1,n)\}\hfill \\ l_3=\{(0,0),(1,\lfloor {\displaystyle \frac{1\times n}{m}}\rfloor ),(2,\lfloor {\displaystyle \frac{2\times n}{m}}\rfloor ),...,(m,\lfloor {\displaystyle \frac{m\times n}{m}}\rfloor )\}\hfill \\ l_4=\{(m,0),(m-1,n-\lfloor {\displaystyle \frac{1\times n}{m}}\rfloor ),(2,\lfloor {\displaystyle \frac{2\times n}{m}}\rfloor ),...,(0,n-\lfloor {\displaystyle \frac{m\times n}{m}}\rfloor )\}\hfill \end{array}\right.\end{array}$$

(15)

be four sets of elements in R (horizontal, vertical, and diagonal element w.r.t. c in R). Let some $k\in \mathbb{N}$ such that $k\le min\{{\displaystyle \frac{m}{8}},{\displaystyle \frac{n}{8}}\}$, and consider

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{v}_1\left(x\right)=(x+k,c_2),\hfill & x\in [0,m-2k]\hfill \\ v_2\left(x\right)=(c_1,x+k),\hfill & x\in [0,n-2k]\hfill \\ v_3\left(x\right)=(x+k,\lfloor {\displaystyle \frac{(x+k)\times n}{m}}\rfloor ),\hfill & x\in [0,m-2k]\hfill \\ v_4\left(x\right)=(m-(x+k),n-\lfloor {\displaystyle \frac{(m-(x+k\left)\right)\times n}{m}}\rfloor ),\hfill & x\in [0,m-2k]\hfill \end{array}\right.\end{array}$$

(16)

Define $V_1,V_2,V_3,V_4$ with same domain as $v_1,v_2,v_3,v_4$ respectively such that

$$\begin{array}{ccc}\hfill V_{\alpha }\left(x\right)=\mathcal{N}(R,i,j,k)& \hfill & \hfill (i,j)=v_{\alpha }\left(x\right)\end{array}$$

(17)

Let some $\delta ,\gamma ,\omega \in \mathbb{R}$ to be the threshold that represents the red value range for the component body and the red value for text on it or other color on it if exist. Consider $V_1^{\prime },V_2^{\prime },V_3^{\prime },V_4^{\prime }$ such that

$$\begin{array}{c}\hfill V_{\alpha }^{\prime }=\left\{\begin{array}{cc}0,\hfill & \delta \le V_{\alpha }\left(x\right)<\gamma \hfill \\ 0,\hfill & \omega \le V_{\alpha }\left(x\right)\hfill \\ 1,\hfill & otherwise\hfill \end{array}\right.\end{array}$$

(18)

Then for $V_1^{\prime },V_2^{\prime },V_3^{\prime },V_4^{\prime }$, let ${\lambda }_{11},{\lambda }_{12},...,{\lambda }_{41},{\lambda }_{42}$ to be the values farthest (but connected) from the center of domain which maps to 0 respectively. For rectangle component, we could have the welding area $A_1,...,A_8$ as follows:

(19)

And for component in other shape such as round, we could have the welding area $A_1,...,A_8$ as follows:

(20)

The welding area matrix $A_i$ is defined to be in same shape with R, but every element outside of the welding area is valued by $-1$.

Similarly, we take some $k,\delta ,\gamma ,\omega$, and then we could find two other sets of $A_1,...,A_8$ for green matrix and blue matrix $G,B$. Denoting the welding area of $R,G,B$ to be $A_1^R,...,A_8^R$, $A_1^G,...,A_8^G$, $A_1^B,...,A_8^B$ respectively. They we say $A_1^{*},...,A_8^{*}$ defined by

$$\begin{array}{cc}\hfill & {\left(a_{\alpha }^{*}\right)}_{i,j}\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill & =\left\{\begin{array}{cc}(R=-1,G=-1,B=-1),\hfill & ifmin\{{\left(a_{\alpha }^R\right)}_{i,j},{\left(a_{\alpha }^G\right)}_{i,j},{\left(a_{\alpha }^B\right)}_{i,j}\}=-1\hfill \\ (R={\left(a_{\alpha }^R\right)}_{i,j},G={\left(a_{\alpha }^G\right)}_{i,j},B={\left(a_{\alpha }^B\right)}_{i,j}),\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}$$

(22)

to be the welding area.

5.4. Refine Welding Area

Now suppose we have $W_1,W_2,...$ to be the welding areas where $W_{\beta }$ is some combination of element in $A_1^{*},...,A_8^{*}$. Since the PCB is in green, we consider function $S_r(M,j),S_c(M,i)$ which take the average of elements with non-negative value in jth row and ith column of matrix M, such that the average is defined by $avg(x_1,...,x_n)=(R={\displaystyle \frac{{\sum }_{i\le n}{x}_{iR}}{n}},G={\displaystyle \frac{{\sum }_{i\le n}{x}_{iG}}{n}},B={\displaystyle \frac{{\sum }_{i\le n}{x}_{iB}}{n}})$ for $x_i=(R=x_{iR},G=x_{iG},B=x_{iB})$

For each welding area, we could change the value of every element in ith column to $(R=-1,G=-1,B=-1)$ if

$$\begin{array}{ccccc}\hfill S_c(R,i)=(R=S_1,G=S_2,B=S_3)& \hfill & \hfill satisfy& \hfill & \hfill \left\{\begin{array}{c}{S}_1\in Gree{n}_R\hfill \\ S_2\in Gree{n}_G\hfill \\ S_3\in Gree{n}_B\hfill \end{array}\right.\end{array}$$

for $Gree{n}_R,Gree{n}_G,Gree{n}_B$ is the color range of the PCB. Similarly, we change value at jth column to $(R=-1,G=-1,B=-1)$ if

$$\begin{array}{ccccc}\hfill S_r(R,i)=(R=S_1,G=S_2,B=S_3)& \hfill & \hfill satisfy& \hfill & \hfill \left\{\begin{array}{c}{S}_1\in Gree{n}_R\hfill \\ S_2\in Gree{n}_G\hfill \\ S_3\in Gree{n}_B\hfill \end{array}\right.\end{array}$$

Taking (5.6) as an example, the process would be as follows:

(23)

5.5. RGB to Normalized Gray Scale

We use the formula $Grayscale=0.299R+0.587G+0.114B$ as discussed in [34] to transform welding area matrix $W_1,...$ into gray scale matrix. For example, if we now let the example discussed in Section 5.2 to be the non-negative part of welding area matrix, we will have the following:

$$\begin{array}{c}\hfill \left[\begin{array}{ccc}R=219,G=48,B=122& & R=103,G=12,B=77\\ R=66,G=200,B=187& & R=0,G=255,B=0\end{array}\right]\end{array}$$

where the element with $(R=-1,G=-1,B=-1)$ is transformed to $-1$

Next, we normalize the gray scale matrix where we let the sum of all non-negative element in the matrix to be 100 by letting the element at ith row jth column in normalized gray scale matrix to be

$$\begin{array}{c}\hfill m_{i,j}^{*}=\left\{\begin{array}{cc}-1,\hfill & m_{i,j}=-1\hfill \\ {\displaystyle \frac{100\times m_{i,j}}{{\sum }_{r,s}max\{m_{r,s},0\}}},\hfill & otherwise\hfill \end{array}\right.\end{array}$$

(24)

where $m_{i,j}$ is the element at ith row jth column in gray scale matrix. In this example, we will have the

$$\begin{array}{c}\hfill \left[\begin{array}{ccc}23.2671& & 10.0838\\ 34.2724& & 32.3767\end{array}\right]\end{array}$$

(25)

6. Experimental Result and Discussion

• (Precision) Sensitivity to slight change in image
• (Efficiency) Computational cost comparison for entire component and only welding part
• (Independency) Do not need large training data sets
• (Directivity) Can provide actionable insights for manufacturing process improvement and product repair

7. Conclusions

References

1. Li, F.; Wu, X.; Phillips, M.A. Technical efficiency of the urban gas industry in china. Environmental Science and Pollution Research 2023, 30, 104477–104488. [Google Scholar] [CrossRef] [PubMed]
2. Huang, L. City natural gas metering. Natural Gas—Extraction to End Use; Gupta, SB, Ed.; InTech: Rijeka, Croatia, 2012; pp. 181–208. [Google Scholar]
3. Zehua, S. Intertet of Things And Smart City; Renmin Unviersity of China Press, 2022.
4. Webster, J.G.; Eren, H. Measurement, Instrumentation, and Sensors Handbook: Two-Volume Set; CRC press, 2018.
5. Sun, Q.; Li, H.; Ma, Z.; Wang, C.; Campillo, J.; Zhang, Q.; Wallin, F.; Guo, J. A comprehensive review of smart energy meters in intelligent energy networks. IEEE Internet of Things Journal 2015, 3, 464–479. [Google Scholar] [CrossRef]
6. Ilea, P.; Săvescu, D.; Stoica, A. Some aspects of gas meter quality and analyzing the sustainability of a gas meter element. In Proceedings of the IOP Conference Series: Materials Science and Engineering. IOP Publishing, 2019, Vol. 568, p. 012063.
7. Wu, R.; Backus, S.; Basu, I.; Blanchard, P.; Brice, K.; Dryfhout-Clark, H.; Fowlie, P.; Hulting, M.; Hites, R. Findings from quality assurance activities in the Integrated Atmospheric Deposition Network. Journal of Environmental Monitoring 2009, 11, 277–296. [Google Scholar] [CrossRef] [PubMed]
8. Hacke, P.; Lokanath, S.; Williams, P.; Vasan, A.; Sochor, P.; TamizhMani, G.; Shinohara, H.; Kurtz, S. A status review of photovoltaic power conversion equipment reliability, safety, and quality assurance protocols. Renewable and Sustainable Energy Reviews 2018, 82, 1097–1112. [Google Scholar] [CrossRef]
9. Kavitha, R.; Akshatha, K. Component identification and defect detection of printed circuit board using artificial intelligence. In Proceedings of the 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC). IEEE. 2023; pp. 100–106. [Google Scholar]
10. Sezer, A.; Altan, A. Optimization of deep learning model parameters in classification of solder paste defects. In Proceedings of the 2021 3rd international congress on human-computer interaction, optimization and robotic applications (HORA). IEEE. 2021; pp. 1–6. [Google Scholar]
11. Chen, I.C.; Hwang, R.C.; Huang, H.C. Pcb defect detection based on deep learning algorithm. Processes 2023, 11, 775. [Google Scholar] [CrossRef]
12. Chen, X.; Wu, Y.; He, X.; Ming, W. A comprehensive review of deep learning-based PCB defect detection. IEEE Access 2023. [Google Scholar] [CrossRef]
13. Ling, Q.; Isa, N.A.M. Printed circuit board defect detection methods based on image processing, machine learning and deep learning: A survey. IEEE Access 2023, 11, 15921–15944. [Google Scholar] [CrossRef]
14. Baygin, M.; Karakose, M.; Sarimaden, A.; Erhan, A. Machine vision based defect detection approach using image processing. In Proceedings of the 2017 international artificial intelligence and data processing symposium (IDAP). Ieee. 2017; pp. 1–5. [Google Scholar]
15. Chang, Y.; Xue, Y.; Zhang, Y.; Ma, J.; Li, G.; Wu, D.; Zhan, Q.; Zuo, J. Research on PCB solder joint defect detection method based on machine vision. In Proceedings of the Fourteenth International Conference on Graphics and Image Processing (ICGIP 2022). SPIE, 2023. Vol. 12705; pp. 181–188.
16. Bhatt, P.M.; Malhan, R.K.; Rajendran, P.; Shah, B.C.; Thakar, S.; Yoon, Y.J.; Gupta, S.K. Image-based surface defect detection using deep learning: A review. Journal of Computing and Information Science in Engineering 2021, 21, 040801. [Google Scholar] [CrossRef]
17. Liu, G. Surface defect detection methods based on deep learning: a brief review. In Proceedings of the 2020 2nd International Conference on Information Technology and Computer Application (ITCA). IEEE. 2020; pp. 200–203. [Google Scholar]
18. Kantorovich, L.V. On the Translocation of Masses. Journal of mathematical sciences 2006, 133. [Google Scholar] [CrossRef]
19. Liu, Z.; Qu, B. Machine vision based online detection of PCB defect. Microprocessors and Microsystems 2021, 82, 103807. [Google Scholar] [CrossRef]
20. Ren, Z.; Fang, F.; Yan, N.; Wu, Y. State of the art in defect detection based on machine vision. International Journal of Precision Engineering and Manufacturing-Green Technology 2022, 9, 661–691. [Google Scholar] [CrossRef]
21. Liao, S.; Huang, C.; Zhang, H.; Gong, J.; Li, M.; Wang, Z. Object detection of welding defects in SMT electronics production based on deep learning. In Proceedings of the 2022 23rd International Conference on Electronic Packaging Technology (ICEPT). IEEE, 2022. pp. 1–5.
22. Legon, A.; Deo, M.; Albin, S.; Audette, M. Detection and classification of PCB defects using deep learning methods. In Proceedings of the 2022 IEEE 31st Microelectronics Design & Test Symposium (MDTS). IEEE, 2022. pp. 1–6.
23. Liu, X.; Wang, J.; Zhang, R.; Ruan, W. Deep learning-based defect detection for printed circuit boards. In Proceedings of the Third International Conference on Computer Vision and Pattern Analysis (ICCPA 2023). SPIE, 2023. Vol. 12754; pp. 897–902.
24. Ling, Z.; Zhang, A.; Ma, D.; Shi, Y.; Wen, H. Deep Siamese semantic segmentation network for PCB welding defect detection. IEEE Transactions on Instrumentation and Measurement 2022, 71, 1–11. [Google Scholar] [CrossRef]
25. Pham, T.T.A.; Thoi, D.K.T.; Choi, H.; Park, S. Defect detection in printed circuit boards using semi-supervised learning. Sensors 2023, 23, 3246. [Google Scholar] [CrossRef] [PubMed]
26. Mémoli, F. Gromov–Wasserstein distances and the metric approach to object matching. Foundations of computational mathematics 2011, 11, 417–487. [Google Scholar] [CrossRef]
27. Colombo, M.; Di Marino, S. Equality between Monge and Kantorovich multimarginal problems with Coulomb cost. Annali di Matematica Pura ed Applicata (1923-) 2015, 194, 307–320. [Google Scholar] [CrossRef]
28. Dantzig, G.B. Linear programming and extensions. In Linear programming and extensions; Princeton university press, 2016.
29. Kuhn, H.W. The Hungarian method for the assignment problem. Naval research logistics quarterly 1955, 2, 83–97. [Google Scholar] [CrossRef]
30. Bertsekas, D.P. The auction algorithm: A distributed relaxation method for the assignment problem. Annals of operations research 1988, 14, 105–123. [Google Scholar] [CrossRef]
31. Evans, L.C. Partial differential equations and Monge-Kantorovich mass transfer. Current developments in mathematics 1997, 1997, 65–126. [Google Scholar] [CrossRef]
32. Banciu, M. Dual simplex. In Wiley Encyclopedia of Operations Research and Management Science; Wiley, 2011.
33. Potra, F.A.; Wright, S.J. Interior-point methods. Journal of computational and applied mathematics 2000, 124, 281–302. [Google Scholar] [CrossRef]
34. Alrubaie, S.H.; Hameed, A.H. Dynamic weights equations for converting grayscale image to RGB image. Journal of University of Babylon for Pure and Applied Sciences 2018, 26, 122–129. [Google Scholar] [CrossRef]