1. Introduction

Figure 1. Challenges of OCR in industrial environments. Optical Character Recognition (OCR) faces several challenges in industrial settings. It includes the ability to recognize text in a variety of sizes, fonts, and orientations, as well as the capability to identify text in images that have low resolution and are affected by noise., such as those with multiple light sources or bright lighting. These difficulties must be overcome to enable accurate and reliable text recognition in industrial environments.

Figure 1. Challenges of OCR in industrial environments. Optical Character Recognition (OCR) faces several challenges in industrial settings. It includes the ability to recognize text in a variety of sizes, fonts, and orientations, as well as the capability to identify text in images that have low resolution and are affected by noise., such as those with multiple light sources or bright lighting. These difficulties must be overcome to enable accurate and reliable text recognition in industrial environments.

2. Related Works

Table 1. Contributions and improvements from related works.

3. The Proposed Method

3.3. Relevant Principles of Image Correction Based on Probabilistic Hough Transform

The projection of the four corner coordinates obtained from the Progressive Probabilistic Hough Transform is presented. The projection process involves mapping a graphical object from one plane to another using a projection matrix. Projection transformations can be divided into two categories: affine transformations and linear transformations. The relationship between the two subsets of projection transformations is shown in Figure 2b.

Linear transformation includes various transformations, including image rotation, skew transformation, scaling, and their combinations. After linear transformation, the origin of the image coordinates remains unchanged, and straight lines retain their straightness and parallel lines their parallelism.

Affine transformation, also known as affine projection, is a combination of linear transformation and translation. In geometry, it refers to the process of transforming one vector space into another by applying a combination of translation and linear transformations.

An equation can be used to illustrate this concept.

Assume K and J are two vector spaces. Defined as follows:

$$\left\{\begin{array}{c}K=(x,y)\\ J=(x^{\prime },y^{\prime })\end{array}\right.$$

(5)

The transformation of the vector space from K to J is given by the following formula:

$$J=K\times \omega +b$$

(6)

The equation will be divided to produce:

$$\left\{\begin{array}{c}{x}^{\prime }={\omega }_{00}\times x+{\omega }_{01}\times y+b_0\\ y^{\prime }={\omega }_{10}\times x+{\omega }_{11}\times y+b_0\end{array}\right.$$

(7)

Matrix multiplication conversion:

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right]=\left[\begin{array}{ccc}{\omega }_{00}& {\omega }_{12}& b_0\\ {\omega }_{10}& {\omega }_{11}& b_1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=M\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(8)

The Probabilistic Hough Transform is used to locate the four corners of an industrial nameplate in the image. A parameter matrix "M" is used to implement geometric transformations such as translation, scaling, rotation and mirroring. M acts as an intermediary between two vector spaces that undergo an affine transformation. Affine transformation is a combination of translation and linear transformation where the vector space is transformed and translated from one space to another.

Our approach study applies a perspective transformation, also known as projection transformation, to turn the irregular square in the 2D image into a rectangular shape. Using the Probabilistic Hough Transform, the four vertices of the nameplate of the equipment are determined. The projection matrix then projects the irregular quadrilateral onto the chosen plane. The projection transformation is completed through three stages.

1. General perspective transformation formula:

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ \omega \end{array}\right]=\left[\begin{array}{c}u\\ v\\ \omega \end{array}\right]\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ {\omega }_{21}& {\omega }_{22}& a_{23}\\ {\omega }_{31}& {\omega }_{32}& a_{33}\end{array}\right]$$

(9)

The parameters u and v, which define $\omega =1$ and $a_{3}3=1$ and the image coordinates $(x,y)$ generated via projection transformation, are taken from the original image. When x and y are satisfied:

$$\left\{\begin{array}{c}x=\frac{x^{\prime }}{{\omega }^{\prime }}\\ y=\frac{y^{\prime }}{{\omega }^{\prime }}\end{array}\right.$$

(10)

2. The eight parameters of the perspective transformation matrix need four coordinates to match to the eight equations that must be resolved. Two particular types of perspective transformation are linear transformation and affine transformation. The converted forms of $x_n$ and $y_m$ satisfy:

$$\left\{\begin{array}{c}{x}_n=\frac{x^{\prime }}{{\omega }^{\prime }}=\frac{a_{11}\times u+a_{21}\times v+a_{31}\times 1}{a_{13}\times u+a_{23}\times v+1}\\ y_m=\frac{y^{\prime }}{{\omega }^{\prime }}=\frac{a_{12}\times u+a_{22}\times v+a_{32}\times 1}{a_{13}\times u+a_{23}\times v+1}\end{array}\right.$$

(11)

Figure 3. Text instances of nameplates and labels are depicted in this image, which represents a typical example from our industrial nameplate dataset. The instances of text are particularly challenging to predict due to their complexity, which is characterized by variations in color, shape, background, and dense with special distributions. These challenging aspects of the dataset require advanced text detection and recognition techniques to achieve accurate results.

Figure 3. Text instances of nameplates and labels are depicted in this image, which represents a typical example from our industrial nameplate dataset. The instances of text are particularly challenging to predict due to their complexity, which is characterized by variations in color, shape, background, and dense with special distributions. These challenging aspects of the dataset require advanced text detection and recognition techniques to achieve accurate results.

3. The solution to the equations provides the necessary eight parameters for the perspective transformation. By multiplication with a projection transformation matrix, the 2D image is converted into a new planar image. It is assumed that the projection center, target point, image point and align in a straight line, and the surface that casts a shadow is rotated around the perspective axis by a specified amount. The target matrix is obtained by preserving the original projection geometry while eliminating the original projection light beam.

4. Experiments & Analysis

4.2. Metrics

When evaluating text recognition models, accuracy, precision, recall and F-measure are commonly used metrics. To fully understand these metrics, it is important to examine the confusion matrix. The confusion matrix is a graphical illustration that shows the association between predicted and actual categories, and it is also referred to as an error matrix or contingency matrix.

The confusion matrix typically has columns representing the predicted categories and rows representing the actual categories. The sum of each column represents the total number of predictions made for each category, while the sum of each row represents the total number of occurrences of each actual category. The numbers in each cell indicate the count of actual data points that were correctly classified as belonging to the corresponding predicted category.

Table 2. Confusion matrix used to form evaluation metrics.

Table 2. Confusion matrix used to form evaluation metrics.

/		Actual class
		Positive	Negative
Predicted class	Positive	True Positive(TP)	False Positive(FP)
	Negative	False Negtive(FN)	True Negtive(TN)

Accuracy: the proportion of correctly predicted samples in all samples; nevertheless, this assessment indicator suffers greatly when the positive and negative samples are out of proportion.

$$Accuracy=\frac{TP+NP}{TP+TN+FP+FN}$$

(12)

Precision: the measure of how well the model identifies true positive cases out of all positive predictions.

$$Precision=\frac{TP}{TP+FP}$$

(13)

Recall: consider the ratio of positively anticipated samples to all positive samples.

$$Recall=\frac{TP}{TP+FN}$$

(14)

A statistic called F-measure is sometimes referred to as F-ScoreHmean. The weighted harmonic average of Precision and Recall is known as F-Measure. It is referred to as the F1 value when $\beta =1$. A higher F1 score indicates improved performance of the experimental approach.

$$F_{measure}=\frac{(1+{\beta }^2)\times Precision\times Recall}{({\beta }^2\times Precision+Recall)}$$

(15)

4.4. Comparative Experiment

The effectiveness of conventional text detection techniques is limited to basic text distribution and lacks robustness, as the spacing between characters must not be too close for traditional text recognition. Two popular text detection algorithms, CTPN [31] and EAST [32], possess limitations, as CTPN can only detect horizontally dispersed text while the EAST method produces incorrect predictions for vertical text instances. Moreover, the unique visual characteristics of warped text remain unresolved by both approaches despite their widespread use. To ensure reliable text detection, we leverage DBNet and SAST [33], both of which exhibit high detection accuracy. Deep learning models often incorporate multiple underlying structures, and ResNet was proposed in 2015 to resolve the bottleneck problem associated with the network depth of classic deep learning networks. While it was generally believed that the effectiveness of a network increased with its depth, research showed that excessively deep networks could worsen the network effect by causing the gradient to vanish. He Kaiming’s ResNet [34,35] effectively addressed this issue. As a result, we have chosen to employ several iterations of ResNet as the underlying network structure for the detection model.

Table 3. The results of a comparative test using DBNet with ResNet18 as the backbone. The performance of models built from different scenarios is not directly transferable to complex industrial situations. Nevertheless, our training with new images has significantly improved model accuracy.

Table 3. The results of a comparative test using DBNet with ResNet18 as the backbone. The performance of models built from different scenarios is not directly transferable to complex industrial situations. Nevertheless, our training with new images has significantly improved model accuracy.

Dataset	Precision	Recall	F-measure	FPS
TD500	0.86	0.79	0.82	66
TD500 trained model in MEND	0.34	0.14	0.84	58
SynthText pre-trained in MEND	0.14	0.01	0.02	61
SynthText pre-trained, and trained in new pictures in MEND	0.84	0.81	0.82	52

Table 4. Comparison test: DBNet takes ResNet50 as the backbone. Using a deeper neural network as the backbone ResNet50 with slightly reduced prediction speed still achieved significantly better performance than ResNet18.

Table 4. Comparison test: DBNet takes ResNet50 as the backbone. Using a deeper neural network as the backbone ResNet50 with slightly reduced prediction speed still achieved significantly better performance than ResNet18.

Dataset	Precision	Recall	F-measure	FPS
TD500	0.91	0.80	0.85	40
TD500 trained model in MEND	0.36	0.26	0.30	28
SynthText pre-trained in MEND	0.06	0.16	0.86	25
SynthText pre-trained, and trained in new pictures in MEND	0.86	0.84	0.85	36

Table 5. Comparative test results of the SAST approach using the ResNet50-D network as a backbone for text recognition in complex industrial scenarios. To optimize its performance, the approach is initially trained on the TOTALTEXT dataset and then fine-tuned on new images. The results show that the SAST approach achieves high accuracy while maintaining appropriate recognition speed.

Table 5. Comparative test results of the SAST approach using the ResNet50-D network as a backbone for text recognition in complex industrial scenarios. To optimize its performance, the approach is initially trained on the TOTALTEXT dataset and then fine-tuned on new images. The results show that the SAST approach achieves high accuracy while maintaining appropriate recognition speed.

Dataset	Precision	Recall	F-measure	FPS
ICDAR2015	0.91	0.83	0.87	12
TD500 pre-trained model in MEND	0.09	0.083	0.08	10
ICDAR2015 pre-trained, and trained in new pictures in MEND	0.72	0.45	0.55	11
TOTALTEXT	0.86	0.84	0.85	25
TOTALTEXT pre-trained model in MEND	0.17	0.08	0.10	25
TOTALTEXT pre-trained and trained in new pictures in MEND	0.91	0.86	0.89	25

Table 6. Comparative experiment: Faster RCNN and YOLOv3. As a comparison for text detection in complex environments, experiments were conducted using Faster-RCNN and Yolov3. First, training was performed on the extraction of features of industrial nameplates separately, and then fine-tuning was carried out on the generated data according to the strategy. The results showed that both methods achieved a slight improvement.

Table 6. Comparative experiment: Faster RCNN and YOLOv3. As a comparison for text detection in complex environments, experiments were conducted using Faster-RCNN and Yolov3. First, training was performed on the extraction of features of industrial nameplates separately, and then fine-tuning was carried out on the generated data according to the strategy. The results showed that both methods achieved a slight improvement.

Dataset	Precision	Recall	F-measure	FPS
Faster-RCNN trained in MEND	0.71	0.73	0.72	14
Faster-RCNN trained in new
pictures in MEND(ours)	0.85	0.75	0.80	14
Yolov3 trained model in MEND	0.72	0.45	0.55	25
Yolov3 trained in new
pictures in MEND (ours)	0.80	0.82	0.81	25

ResNet18 has fast convergence time and high accuracy performance compared to other residual network models. A deeper model may yield better results due to its ability to extract more complex features by increasing the network depth. ResNet50, which has a more complex networked structure than ResNet18, offers improved accuracy without showing any noticeable degradation. For the differentiable binarization approach, we used both ResNet18 and ResNet50 as the basic networks. In addition, we used the enhanced 50-layer residual network ResNet50-D [35] as the basis for the SAST detector. This was done to speed up the subsequent training phase.

We conducted experiments on three different datasets, namely MSRA-TD500, ICDAR2015, and TOTALTEXT, with the aim of training the models immediately and collecting evaluation data. To increase the model’s resilience, a novel dataset was generated by utilizing the gathered data and the rectified training images. During the pre-training phase, three separate datasets and two different detectors were used to achieve better weighting and detection speed. We performed over 5,000 iterations on all pre-trained models, with about 200 images transformed into different perspectives for the new dataset.

We evaluated the performance of DBNet using ResNet18 and ResNet50 as backbones on the MSAR-TD500 and MEND datasets. As shown in the tabular data, it is not feasible to directly apply the models to complex industrial scenarios. As part of our evaluation of the SAST approach, we conducted a comparative experiment using the ResNet50-D enhanced residual network as the backbone for our system, with the same dataset to ensure a fair comparison. Specifically, we evaluated our method’s efficacy and the difficulty level of the dataset using the ICDAR2015 and TOTALTEXT datasets. The results of the experiment, including precision, recall, and F-measure, are presented in Table 4. Our pre-trained and re-trained models demonstrated excellent performance on these datasets. However, it should be noted that text identification algorithms trained on other datasets may not provide optimal results when used in complex industrial environments. This is because, in addition to the correctness of the character set, line-level text samples must also be collected to address the challenges presented by such settings.

The method proposed in this paper is a pre-processing technique for text detection, which does not affect the efficiency of the model’s use, but can enhance the inference accuracy. In practical applications, this pre-processing algorithm has a relatively small computational cost and high operating efficiency on the experimental host, with a speed of up to 20fps. The delay caused by this method during barcode scanning or data collection is negligible for practical applications. Our method has also effectively improved the detection accuracy in experiments with Faster RCNN and YOLOv3 Table 6. However, since these two detection algorithms are general-purpose and not specifically optimized for text detection, their accuracy is lower than that of other algorithms

5. Conclusions

References

1. Yu, X.; Ye, X.; Zhang, S. Floating pollutant image target extraction algorithm based on immune extremum region. Digital Signal Processing 2022, 123, 103442. [Google Scholar] [CrossRef]
2. Yu, X.; Tian, X. A fault detection algorithm for pipeline insulation layer based on immune neural network. International Journal of Pressure Vessels and Piping 2022, 196, 104611. [Google Scholar] [CrossRef]
3. Liang, T.; Bao, H.; Pan, W.; Pan, F. Traffic sign detection via improved sparse R-CNN for autonomous vehicles. Journal of Advanced Transportation 2022, 2022, 1–16. [Google Scholar] [CrossRef]
4. Liao, M.; Wan, Z.; Yao, C.; Chen, K.; Bai, X. Real-time scene text detection with differentiable binarization. Proceedings of the AAAI conference on artificial intelligence, 2020, Vol. 34, pp. 11474–11481.
5. Sabu, A.M.; Das, A.S. A Survey on various Optical Character Recognition Techniques. 2018 conference on emerging devices and smart systems (ICEDSS). IEEE, 2018, pp. 152–155.
6. Wu, W.; Xing, J.; Yang, C.; Wang, Y.; Zhou, H. Texts as Lines: Text Detection with Weak Supervision. Mathematical Problems in Engineering 2020, 2020, 1–12. [Google Scholar] [CrossRef]
7. Li, J.; Huang, T.; Yang, Y.; Xu, Q. Detection and Recognition of Characters on the Surface of Metal Workpieces with Complex Background. 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), 2020.
8. Yuan, J.; Guo, M.; Huang, B.; Hu, R.; Dian, S. Processing and Recognition of Characters Image in Complex Environment. 2022 International Conference on Innovations and Development of Information Technologies and Robotics (IDITR), 2022, pp. 100–104.
9. Khan, T.; Sarkar, R.; Mollah, A.F. Deep learning approaches to scene text detection: a comprehensive review. Artificial Intelligence Review 2021, 54, 3239–3298. [Google Scholar] [CrossRef]
10. Long, S.; He, X.; Yao, C. Scene text detection and recognition: The deep learning era. International Journal of Computer Vision 2021, 129, 161–184. [Google Scholar] [CrossRef]
11. Chen, X.; Zhang, Z.; Qiao, Y.; Lai, J.; Jiang, J.; Zhang, Z.; Fu, B. Orientation robust scene text recognition in natural scene. 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2019, pp. 901–906.
12. Baifeng, N.; Ganzi, H.; Yu, Y. Research on nameplate image recognition algorithm based on R-CNN and SSD deep learning detection methods. 2022 IEEE 2nd International Conference on Power, Electronics and Computer Applications (ICPECA). IEEE, 2022, pp. 580–584.
13. Shuliang, Z.; Xin, H.; Haochen, Z.; Jianyu, W. Design of text position detection method for electrical equipment nameplate. 2021 International Conference on Cyber-Physical Social Intelligence (ICCSI). IEEE, 2021, pp. 1–4.
14. Panhwar, M.A.; Memon, K.A.; Abro, A.; Zhongliang, D.; Khuhro, S.A.; Memon, S. Signboard detection and text recognition using artificial neural networks. 2019 IEEE 9th international conference on electronics information and emergency communication (ICEIEC). IEEE, 2019, pp. 16–19.
15. Wu, Y.; Li, Z.; Wang, Y.; Huang, Z.; Zheng, Z. Application Research of Feature Extraction Method of Power Equipment Nameplate. Computer Science and Application 2019, 09, 2084–2097. [Google Scholar] [CrossRef]
16. Zhao, K.; Han, Q.; Zhang, C.B.; Xu, J.; Cheng, M.M. Deep hough transform for semantic line detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 2021, 44, 4793–4806. [Google Scholar] [CrossRef] [PubMed]
17. Kagawa, T.; Ikemoto, M.; Ohtake, S. A robust method of IC seating inspection in burn-in sockets using Hough transform. 2022 IEEE International Conference on Consumer Electronics-Taiwan. IEEE, 2022, pp. 1–2.
18. Aslani, A.A.; Zolfaghari, M.; Sajedi, H. Automatic Counting Red Blood Cells in the Microscopic Images by EndPoints Method and Circular Hough Transform. 2022 16th International Conference on Ubiquitous Information Management and Communication (IMCOM). IEEE, 2022, pp. 1–5.
19. Marzougui, M.; Alasiry, A.; Kortli, Y.; Baili, J. A lane tracking method based on progressive probabilistic Hough transform. IEEE access 2020, 8, 84893–84905. [Google Scholar] [CrossRef]
20. Kumar, S.; Jailia, M.; Varshney, S. An efficient approach for highway lane detection based on the Hough transform and Kalman filter. Innovative infrastructure solutions 2022, 7, 290. [Google Scholar] [CrossRef]
21. Ahmad, R.; Naz, S.; Razzak, I. Efficient skew detection and correction in scanned document images through clustering of probabilistic hough transforms. Pattern Recognition Letters 2021, 152, 93–99. [Google Scholar] [CrossRef]
22. Ma, C.; Li, X.; Li, Y.; Tian, X.; Wang, Y.; Kim, H.; Serikawa, S. Visual information processing for deep-sea visual monitoring system. Cognitive Robotics 2021, 1, 3–11. [Google Scholar] [CrossRef]
23. Yang, S.; Lu, H.; Li, J. Multifeature fusion-based object detection for intelligent transportation systems. IEEE Transactions on Intelligent Transportation Systems 2022. [Google Scholar] [CrossRef]
24. Lai, J.; Guo, L.; Qiao, Y.; Chen, X.; Zhang, Z.; Liu, C.; Li, Y.; Fu, B. Robust text line detection in equipment nameplate images. 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2019, pp. 889–894.
25. Li, J.; Zhang, W.; Han, R. Application of machine vision in defects inspection and character recognition of nameplate surface. 2014 13th International Symposium on Distributed Computing and Applications to Business, Engineering and Science. IEEE, 2014, pp. 295–298.
26. Nakayama, Y.; Lu, H.; Li, Y.; Kamiya, T. WideSegNeXt: semantic image segmentation using wide residual network and NeXt dilated unit. IEEE Sensors Journal 2020, 21, 11427–11434. [Google Scholar] [CrossRef]
27. Gupta, A.; Vedaldi, A.; Zisserman, A. Synthetic data for text localisation in natural images. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2315–2324.
28. Yao, C.; Bai, X.; Liu, W.; Ma, Y.; Tu, Z. Detecting texts of arbitrary orientations in natural images. 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012, pp. 1083–1090.
29. Karatzas, D.; Gomez-Bigorda, L.; Nicolaou, A.; Ghosh, S.; Bagdanov, A.; Iwamura, M.; Matas, J.; Neumann, L.; Chandrasekhar, V.R.; Lu, S.; others. ICDAR 2015 competition on robust reading. 2015 13th international conference on document analysis and recognition (ICDAR). IEEE, 2015, pp. 1156–1160.
30. Ch’ng, C.K.; Chan, C.S. Total-text: A comprehensive dataset for scene text detection and recognition. 2017 14th IAPR international conference on document analysis and recognition (ICDAR). IEEE, 2017, Vol. 1, pp. 935–942.
31. Tian, Z.; Huang, W.; He, T.; He, P.; Qiao, Y. Detecting text in natural image with connectionist text proposal network. Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer, 2016, pp. 56–72.
32. Zhou, X.; Yao, C.; Wen, H.; Wang, Y.; Zhou, S.; He, W.; Liang, J. East: an efficient and accurate scene text detector. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 5551–5560.
33. Wang, P.; Zhang, C.; Qi, F.; Huang, Z.; En, M.; Han, J.; Liu, J.; Ding, E.; Shi, G. A single-shot arbitrarily-shaped text detector based on context attended multi-task learning. Proceedings of the 27th ACM international conference on multimedia, 2019, pp. 1277–1285.
34. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
35. He, T.; Zhang, Z.; Zhang, H.; Zhang, Z.; Xie, J.; Li, M. Bag of tricks for image classification with convolutional neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 558–567.