Validating the Use of Mixed Reality in Industrial Quality Control: A Case Study

1. Introduction

Quality control (QC) is the bedrock of industrial manufacturing and is intrinsically tied to operational efficiency, product reliability, and overall customer satisfaction [1,2,3]. In the industry, the criticality of QC is amplified due to the inherent complexity of manufacturing processes, the diversity of parts and components, and the high-stakes safety implications of the final product [4]. As such, the ability to perform quality inspections rapidly and accurately can significantly affect an organization’s bottom line and influence the larger industry’s trajectory.

Defects, especially those detected late in the manufacturing process or after the product’s release, can lead to high costs, including reworking, recalls, or even reputational damage to the manufacturer [1]. Consequently, thorough inspections during various stages of production can significantly reduce these costs, maximizing productivity and profitability. Moreover, a robust quality control system fosters customer confidence in the brand, driving increased sales and maintaining a competitive edge in the market [5,6].

Traditionally, quality inspections in industry have been manual processes, relying heavily on the experience and judgment of human inspectors [7,8]. However, the increasingly intricate nature of modern products, combined with the growing demand for efficiency and productivity, has pushed these traditional inspection methods to their limits. Manual inspections can be time-consuming, prone to errors, and may struggle to detect subtle or complex defects [9]. Furthermore, the potential for inconsistency among inspectors poses a significant challenge to maintaining uniform quality standards [10,11].

In the quest for greater efficiency and precision, industries worldwide are turning to technology. In particular, Mixed Reality (MR), characterized by the enhancement of the real world with digital information, has emerged as a promising solution to these QC challenges. MR allows for real-time, interactive overlays of data in the user’s field of view, potentially revolutionizing inspection processes [12].

MR, when integrated into quality control processes, offers a range of advantages. First, it increases efficiency [13,14,15,16]. By overlaying digital information directly onto real-world components, MR allows inspectors to rapidly identify and assess parts without the need for time-consuming manual checks against plans or specifications. This can significantly speed up the inspection process, allowing for quicker detection of defects and faster correction of production issues.

Also, MR improves accuracy [16]. By leveraging sophisticated imaging and recognition technology, MR can detect defects that may be missed by the human eye. It can also ensure consistency in inspections by applying the same precise standards to every part, every time, eliminating the variability that can occur with human inspections.

The MR technology also opens the possibility of real-time tracking of inspection cycles and individual tasks. This additional feature can provide valuable insights into the efficiency of the manufacturing process, enabling managers to identify bottlenecks and improve workflow designs [17,18].

Understanding cycle times and individual task times is paramount in industrial environments like automotive manufacturing. The time an operator spends on a task or the cycle time of a specific process can provide deep insights into the operational efficiency and productivity of the manufacturing process [16,19]. Moreover, this information, when systematically captured and analyzed, can reveal patterns, inconsistencies, and bottlenecks that might otherwise go unnoticed.

However, despite the potential benefits, the integration of MR into industrial quality control processes is still in its early stages, with many questions remaining about its practical feasibility and effectiveness. Motivated by the pressing need for more efficient and reliable quality control methods and inspired by the potential of MR to transform industrial inspections, this study embarked on an exploratory journey to investigate how MR can be effectively used in industry QC.

The primary objective of this study is to develop an MR-based quality control system using smart glasses, integrated with a server-based image recognition system, and to evaluate this system’s effectiveness in a controlled environment.

This study also intends to evaluate the performance and viability of this system under real world conditions. The objective was to identify possible problems, possibilities for improvements and verify the feasibility of implementing this MR system on a larger operational scale in the manufacturing industry.

In doing so, it is intended to provide a practical roadmap for the adoption of MR in industrial quality control and to contribute to the understanding and advancement of this technology.

The remainder of this paper is organized as follows. The next section provides an overview of object detection. Section 3 details the proposed methodology. Section 4 presents the pilot test carried out in a controlled environment where the intention is to evaluate the performance of the system, identify possible problems and evaluate its scalability.

Section 5 concludes the paper by summarizing the essential findings and highlighting the significance of the proposed methodology in improving quality control system.

2. Object Detection

In the realm of quality control processes enhanced by mixed reality, object detection assumes a fundamental and pivotal role. Object detection serves as the cornerstone of this convergence, seamlessly integrating real-world and digital environments to facilitate meticulous quality assessment and assurance.

Mixed reality, which blends elements of both physical reality and virtual environments, demands a robust mechanism to identify, localize, and analyze objects within the merged context. Object detection technologies, often harnessed through advanced computer vision techniques, enable the system to precisely discern and categorize relevant objects of interest.

The efficacy of quality control processes hinges on accurate and timely identification of specific items, defects, or anomalies. Object detection, operating at the heart of the framework, empowers the system to perform these functions with precision. By seamlessly detecting objects in real-time and overlaying relevant digital information, such as annotations or diagnostic insights, the quality control process achieves an elevated level of comprehensiveness and accuracy.

Over the years, the development of object detection algorithms has undergone significant evolution, driven by advancements in deep learning and the need for more accurate and efficient solutions. One of the pivotal distinctions in this evolution lies in the categorization of algorithms into two-stage and one-stage approaches [20,21].

2.1. Two-Stage Object Detection Algorithms

The inception of two-stage object detection algorithms marked a transformative shift in the field. This paradigm introduced a clear separation between region proposal and object classification stages, addressing the challenges posed by computational inefficiency in previous methods [22]. The R-CNN (Region-based Convolutional Neural Network) family of algorithms, including R-CNN, Fast R-CNN, and Faster R-CNN, played a pivotal role in establishing this approach.

R-CNN, introduced by Girshick et al. in 2014 [23], presented a novel strategy by proposing regions of interest (RoIs) using selective search and then classifying objects within these regions using a convolutional neural network (CNN). This method demonstrated promising results in terms of accuracy but suffered from slow processing speeds due to the sequential nature of region proposal and classification. The subsequent refinement, Fast R-CNN, merged the region proposal and feature extraction stages, sharing computation across different regions and leading to improved efficiency [24].

The pinnacle of the two-stage approach was reached with the introduction of Faster R-CNN by Ren et al. in 2015 [25]. This approach integrated the region proposal process into the network architecture itself using a Region Proposal Network (RPN), enabling end-to-end training. Faster R-CNN achieved remarkable accuracy while significantly reducing processing time, making it a cornerstone in the field of object detection [26,27].

2.2. One-Stage Object Detection Algorithms

While two-stage algorithms were achieving remarkable accuracy, the computational overhead associated with region proposal hindered their real-time applicability. This limitation spurred the development of one-stage object detection algorithms, which aimed to address this issue by performing object classification and localization in a single pass.

The YOLO (You Only Look Once) family of algorithms and the SSD (Single Shot MultiBox Detector) algorithm exemplify the one-stage approach. YOLO, introduced by Redmon et al. in 2016 [28], divided the input image into a grid and predicted bounding boxes and class probabilities directly from grid cells. This design drastically reduced computation time, making real-time object detection feasible [29,30]. YOLO demonstrated competitive accuracy, but its weak point was the challenge of detecting small objects accurately due to the inherent nature of its grid-based architecture.

On the other hand, SSD, introduced by Liu et al. in 2016 [31], adopted a different strategy. It utilized a set of predefined default bounding boxes with varying aspect ratios and sizes, predicting object classes and offsets for these boxes. This design allowed SSD to effectively address the challenge of detecting objects at different scales and improved its accuracy in small object detection. However, SSD still struggled to match the accuracy of two-stage approaches for larger and more complex scenes.

2.3. Evolutionary Trends

The evolution of object detection algorithms has been a dynamic interplay between the two-stage and one-stage approaches, each catering to specific requirements and challenges. While two-stage algorithms excelled in accuracy but often sacrificed speed, one-stage algorithms offered real-time capabilities at the expense of accuracy, especially for small objects and complex scenes [32,33].

Recent advancements have further refined these paradigms. The YOLO series, culminating in YOLOv8 in 2023, showcased significant improvements in accuracy and speed, making them increasingly competitive with two-stage approaches [34,35,36].

Figure 1 presents a concise timeline of the historical dates corresponding to various versions of the YOLO algorithm. The visual representation highlights the progression of YOLO from its inception to the latest versions, including YOLOv8.

In the historical evolution of the YOLO algorithm, certain versions have stood out due to their increased protagonism and importance in shaping the landscape of object detection. Notably, YOLOv1, YOLOv3, and YOLOv4 have garnered significant attention and impact within the computer vision community.

The inaugural version of YOLO introduced the concept of one-stage object detection, revolutionizing the field. Its pioneering design, which predicted bounding boxes and class probabilities directly in a single pass, laid the foundation for subsequent iterations. Despite its relative simplicity, YOLOv1 showcased the potential of real-time object detection, setting the stage for further advancements.

YOLOv3 marked a substantial leap in terms of accuracy and versatility. With the introduction of a feature pyramid network and multiple detection scales, YOLOv3 improved detection performance across various object sizes. This version’s increased accuracy and the ability to handle small and large objects alike further solidified YOLO’s reputation as a powerful object detection algorithm [37].

YOLOv4 represented a significant milestone by substantially improving both accuracy and speed. The architecture was optimized with advanced techniques. This version achieved state-of-the-art performance by enhancing feature extraction, introducing novel training strategies, and incorporating efficient design choices. YOLOv4’s holistic approach to accuracy and speed propelled it to the forefront of object detection algorithms [38,39].

While this version may not be as widely recognized as its predecessors, YOLOv8 has gained prominence due to its reported advancements in accuracy. With the promise of achieving the highest accuracy in the YOLO series, YOLOv8 signifies ongoing efforts to refine the algorithm. This version demonstrates the continued commitment to enhancing object detection performance, particularly in scenarios involving small objects and complex scenes [36].

The YOLOv8 version presents distinct advantages over other solutions in the field of object detection. Its advanced feature extraction capabilities coupled with heightened accuracy ensure precise identification of objects in various scenarios. This version’s holistic approach to accuracy and speed positions it as a standout choice for applications requiring both precision and efficiency in object detection [40,41,42].