Preprint
Article

Digital Twins and Data-driven in Plant Factory: An Online Monitoring Method for Vibration Evaluation and Transplanting Quality Analysis

Altmetrics

Downloads

162

Views

27

Comments

0

A peer-reviewed article of this preprint also exists.

Submitted:

08 May 2023

Posted:

09 May 2023

You are already at the latest version

Alerts
Abstract
Plant Factory Transplanter (PFT) is a key component of the plant factory system. Its operation status directly affects the quality and survival rate of planted seedlings, which in turn affects the overall yield and economic efficiency. To monitor the operation status and transplanting quality of a transplanting machine in a timely manner, the primary task is to use a computerized and easy-to-use method to monitor the transplanting units. Inspired by the latest developments in augmented reality and robotics, a DT model-based and data-driven online monitoring method for plant factory transplanting equipment is proposed. First, a data-driven and virtual model approach is combined to construct a multi-domain digital twin of the transplanting equipment. Then, taking the vibration frequency domain signal above the transplanting manipulator and the image features of the transplanting seedling tray as input variables, the evaluation method and configuration method of the PFT digital twin system are proposed. Finally, the effect of the transplanter is evaluated, and the cycle can be repeated to optimize the transplanter to achieve optimal operation parameters. The results show that the digital twin model can effectively use the sensor data to identify the mechanical vibration characteristics and avoid affecting transplanting quality due to mechanical resonance. At a transplanting rate of 3000 plants/h, the transplanting efficiency can be maintained at a high level and the vibration signal of X, Y, Z-axis above the transplanting manipulator is relatively calm. In this case, Combined the optimal threshold method with the traditional Wiener algorithm, the identification rate of healthy potted seedlings can reach 94.3%. Through comprehensively using the optimal threshold method and 3D block matching filtering algorithm for image threshold segmentation and denoising, the recognition rate of healthy seedlings has reached over 96.10%. In addition, the developed digital twin can predict the operational efficiency and optimal timing of the detected transplanter, even if the environmental and sensor data are not included in the training. The proposed digital twin model can be used for damage detection and operational effectiveness assessment of other plant factory equipment structures.
Keywords: 
Subject: Engineering  -   Mechanical Engineering

1. Introduction

The production of greenhouse vegetables is becoming increasingly industrialized due to serious challenges related to food security, safety, sustainability, and health. (Wang & Zubko et al., 2022). Greenhouses are evolving into high-tech plants characterized by mass production and extensive applications of technologies (van Henten, 2019). Smart horticulture technologies, represented by plant factories that integrate multiple production elements and technical equipment, are changing and upgrading traditional agricultural production methods (Liu et al., 2022; Zhang et al., 2022). However, the intricate planting structure in plant factories poses a considerable challenge to the monitoring of production operations (Sun et al., 2022). As a key technical equipment of plant factories, the operational status of vegetable plant transplanters directly affects the quality and survival rate of the planted seedlings, which also in turn affects the overall yield and economic efficiency (Rahul et al., 2019). A monitoring system that can be applied to the plant factory environment and display the working status of transplanters in real time is needed to design.
There are a large number of sensors and artificial intelligence technologies used for the monitoring of transplanting equipment for agricultural production (Jin et al., 2022; Jin et al., 2020; Tong et al., 2022; Yang et al., 2021), visual localization (Fu et al., 2022; Zhang & Li, 2022), and fault diagnosis (Ji et al., 2021; Ji et al., 2020). These works have greatly advanced the development of transplanting equipment technology. The means of monitoring agricultural equipment have become increasingly intelligent and data-driven, including cloud computing, Internet of Things, big data, and machine learning, augmented reality, and robotics (Wang & Vladislav et al., 2022; Wang et al., 2023). Smart detection technologies integrating virtual reality and physical sensing networks provide new ideas for transplant monitoring (Ariesen-Verschuur et al., 2022; Zhao et al., 2022). A powerful driver for this development is the digital twin. (Pylianidis et al., 2021; Verdouw et al., 2021).
An DT is a virtual representation of a physical entity created digitally. It can simulate the conduct of a bodily entity in a actual surroundings with the assist of data, and add or prolong new skills to the bodily entity by way of capacity of virtual-real interaction feedback, statistics fusion analysis, and iterative optimization of choices (Tao et al., 2018; Tao et al., 2017). Many internationally renowned scholars and companies are exploring the use of digital twin technology in product design, manufacturing, and maintenance. (Luo et al., 2020; Tao et al., 2019; Verdouw et al., 2021). The National Aeronautics and Space Administration (NASA) has investigated a digital twin-based approach for fault monitoring and elimination in complex systems, and applied it to the health management of flight systems. Using the concept of digital twin workshop, Tao Fei et al. designed the composition and operation mechanism of DTS. (Tao et al., 2019). Zhuang Cunbo et al. proposed the architecture and implementation path of the product digital twin (Zhuang et al., 2017), and pointed out that the emergence and development of digital twin technology cannot only provide clear new ideas, but also methods and implementation paths for realizing information-physical systems. The advantages of digital twin consist of decreasing manufacturing time and costs, hiding the complexity of integrating heterogeneous technologies, developing a safer working environment, and setting up extra environmentally sustainable operations (Luo et al., 2020). When applied to factory farming production, digital twin technologies can significantly increase greenhouse greenhouse productivity and sustainability.
The software of digital twin technological know-how in the purposes of plant monitoring has quite a few foreseeable advantages: (1) Growers are capable to remotely reveal and manipulate operations based totally on (near) real-time digital information, alternatively than direct commentary and guide duties in the field; (2) If any (expected) troubles arise, they are alerted and can test the greenhouse in the back of the desk or smartphone; (3) Growers can simulate the have an effect on of corrective and preventive measures on the digital representation. Finally, the grower can function the chosen intervention remotely, and once more a digital illustration can be used to confirm that the (expected) trouble has been solved. (4) This sensible administration cycle will come to be an increasing number of autonomous, requiring no guide intervention from the grower.
An on-line monitoring approach primarily based on DT fashions and information is investigated for plant transplanting gear to acquire dependable and correct monitoring. The relaxation of the paper is prepared as follows: in Section 2, the twin model, comparison technique and configuration approach for PFT monitoring are presented; Section three gives the effects and dialogue of the utility of the PFT monitoring model, and Section four offers the conclusion.

2. Materials and Methods

This case focuses on evaluating the vibration and effect of transplanting manipulator, which plays an intelligent decision-making role in the regulation of transplanting system. PFT’s transplanting robot plays a crucial role in accuracy and quality, and its condition directly affects them. Vibration of transplanting manipulators decreases dimensional accuracy but also shortens the machine’s service life due to fatigue deformation. Besides decreasing dimensional accuracy, vibration of transplanting manipulators also shortens the machine’s service life due to fatigue deformation. To this end, a key component of transplanters, a hybrid DT-based application is investigated for real-time monitoring and vibration evaluation of the manipulator. To evaluate the effect of transplanting, the machine vision and artificial intelligence technology are combined to discriminate and classify the video image information after real-time transplanting, aiming to provide a data support for the wise decision-making of the transplanter, as shown in Figure 1.
Figure 1. A digital twin and application of it to PFT.
Figure 1. A digital twin and application of it to PFT.
Preprints 73037 g001

2.1. Framework

As shown in Figure 2, the hybrid method framework is proposed. To obtain more accurate monitoring results, this framework combines data-driven and model-based approaches. Based on the material properties and operating conditions, a multidomain model of the PFT is developed. In the same way that virtual sensing allows us to calculate the internal state of systems wherever we are, multiphysics field simulations allow us to map boundary conditions to the physical PFT and calculate it wherever we want . Calculating the system state theoretically using simulated internal system values is done by converting the DT physical degradation model into a system state space model . Sensors are mounted on the PFT in a specific manner, and then data is provided to support a data-driven approach to examine the effectiveness of a plant performance evaluation.
In the data-driven approach, a number of steps need to be undertaken in order to transform historical sensory data into useful monitoring data, including data processing, feature extraction from twin data, and feature fusion from twin data . Systematic observation of PFT is conducted using the transplanting stenography monitored by the data-driven approach . Using a hybrid proximity algorithm, the system observations, the system state space model, and the simulated intra-system values are combined . Based on a priori knowledge and simulations of intra-system values, the state of the PFT is monitored. As a result, the monitored states are modified based on observations of the system.
Figure 2. A digital twin-based hybrid approach to monitoring and maintenance framework.
Figure 2. A digital twin-based hybrid approach to monitoring and maintenance framework.
Preprints 73037 g002

2.2. Multi-domain model implementation of plant factory transplanting system

As part of a data-driven algorithmic model, the DT model represents the physical PFT numerically. In the construction of a model, the multi-domain physical fields such as mechanical, electrical, hydraulic, and thermodynamic should be taken into account simultaneously . Software that supports multi-domain modeling includes Unity 3D, SolidWorks 2015, and 3dsMax 2017. Object models at the subsystem level, such as manipulators, cavity seedlings, and cavity seedling trays, can be built by these programs and incorporated into a multidomain system model.
DT is widely recognized as the most reliable and accurate method of monitoring sexual maintenance due to its high fidelity . The accuracy of DT can be improved by obtaining actual operation results (experiments ). Physical PFT-based multi-domain models can also be simulated to obtain simulation results. Model parameters (e.g. material properties, operating conditions) should be the same as the actual parameters. DT consistency is then determined by comparing the experimental and simulation results, an iterative process, as shown in Figure 4. Multi-domain models are modified and simulated iteratively until the difference between simulations and experiments is small. Multi-domain models with small experimental errors can be considered high-fidelity models .
Figure 3. Implementation of the multi-domain model.
Figure 3. Implementation of the multi-domain model.
Preprints 73037 g003
Figure 4. Methods to verify DT accuracy.
Figure 4. Methods to verify DT accuracy.
Preprints 73037 g004
Furthermore, a real-time mapping interface should be built so the DT model can be updated . Modeling involves mapping domain knowledge and operating conditions through interfaces . An expert’s domain knowledge is mainly derived from his or her experience, mechanical manuals, sensor data, and controller parameters. Fault prognosis and monitoring of the tools will be applied on the groundwork of some priori expertise and experience.
The testbed design and sensor installation are shown in Figure 5. The testbed experimental platform includes the data twin service system, the sensing and signal acquisition system, and the PLC control system. The data twin service system contains a virtual entity and a data twin service interface. The sensing and signal acquisition system consists of three-axis acceleration sensor, industrial camera, position sensor, multi-channel signal collector and human-computer interaction interface. The PLC control system consists of manipulator control box and transplanting truss control box. Among them, a three-axis acceleration sensor is installed on the platform above the transplanting manipulator to continuously collect the vibration data of the platform above the manipulator. An industrial camera is mounted on the truss to take images of the entire tray of seedlings after each tray is transplanted. The position sensor is mounted on the metal profile on the side of the conveyor belt, which controls the conveyor belt stepper motor to stop when the seedling tray moves to the position to be transplanted, completing the subsequent transplanting. The main parameters of this experiment are: the working environment is 22°C; the spindle speed is 2300 RPM (revolutions per minute); the horizontal module feed speed is 200 mm/min; the vertical module feed speed is 200 mm/min; the cavity tray conveyor is rated at 200 W; the transmission speed is 0.5 m/s, and the sampling frequency is 50Hz.
Figure 5. Overall design of PFT digital twin experiment platform.
Figure 5. Overall design of PFT digital twin experiment platform.
Preprints 73037 g005
Currently, it is impractical to construct a DT mannequin that thoroughly displays each and every element of the complete system. It is quintessential to purposefully construct a goal-oriented DT mannequin and simplify it. The essential goal of this learn about is to precisely consider the ETQ of plant transplanting. The plant seedlings, pots and transplanting clamping jaws has been simplified somewhat.
Figure 6. Construction of the virtual entity model.
Figure 6. Construction of the virtual entity model.
Preprints 73037 g006
Figure 7. DT model update method for plant transplanting.
Figure 7. DT model update method for plant transplanting.
Preprints 73037 g007
Real-time updates of the DT are vital to hold a proper mapping of the bodily mannequin of the plant transplanter. The foundation for adjustment of the transplanting end-effector manipulator for the duration of the transplanting procedure can be divided into two types:
(a) Transplanting conditions: the transplanting parameters such as spindle motor speed, vibration sign above the transplanting manipulator, and role signal. These parameters are examine from the PFT controller and have a direct have an impact on on the simulation effects of the transplanting process.
(b) Evaluation of the transplanting effect: the evaluation and assessment of the picture sign of the transplanting seedling tray for the duration of the transplanting process, and performs a function of remarks adjustment to alter the parameters of the transplanting platform.

2.3. Data-driven model implementation and dataset description

In the construction of data-driven model, various sensors monitor conditions of PFT are used to collect big data. After collecting historical data, diagnostic and monitoring algorithms are designed and trained. As shown in Figure 8, many data processing methods are required to build the data-driven model, including noise reduction, pre-processing, feature extraction, and feature selection, which requires expert domain knowledge as a foundation.
Figure 8. Construction of the data-driven model.
Figure 8. Construction of the data-driven model.
Preprints 73037 g008
According to setting sensors on the PFT, it is possible to detect and monitor its operating parameters, primarily the state of the PFT and the surrounding environment. From the collected data, feature recognition is performed to identify health-related features. To improve the model training speed and monitoring accuracy, only features strongly related to device health are selected during the feature selection stage. In the construction stage of the algorithm model, data-driven monitoring maintenance of PFT is achieved by constructing fault diagnosis and fault monitoring algorithms based on identified and extracted features.
Due to sensor drift caused by temperature changes, the raw data contains trend items that affect eigenvalue monitoring results. In addition, the sampled raw data are often superimposed with noisy signals, such as industrial frequency signals, periodic interference signals and random interference signals, resulting in burr in the signal waveform. Additionally, raw data smoothing is performed to reduce interference signals and improve vibration curve smoothness. The trend term elimination is performed by a polynomial based on the least squares method, as shown in Figure 9.
Figure 9. Elimination of trends using polynomials of different orders.
Figure 9. Elimination of trends using polynomials of different orders.
Preprints 73037 g009
The total data set collected by the digital twin system of plant transplanting M = {M1, M2, M3, M4}, where M1, M2, M3, and M4 are the data collected under different operating conditions when the equipment is running for 20s, 40s, 60s, and 80s, respectively, as shown in Table 1. Each group contains 256 data files. To ensure the synchronization between multiple sources, the data in the same time period should be input into the digital twin system. The data files are intercepted within 20-80 s from the middle moment, i.e., the vibration and video signals are intercepted and uploaded as data points separately. The vibration signals are stacked as 6-channel samples, and the data dimension of the vibration signals of X,Y, Z channels in the same time period is 128×2048×6, and 15% of the data are selected as the test set.
Table 1. Datasets of transplanting digital twin system.
Table 1. Datasets of transplanting digital twin system.
Work condition serial number Working Status Type of data Acquisition frequency
1 Standby
Start-up
Vibration D1 2560Hz
Transverse motor power D2 50Hz
Vertical motor power D3 50Hz
Parallel motor power D4 50Hz
2 Low speed transplant
1500 plants/h
Vibration D1 2560Hz
Transverse motor power D2 50Hz
Vertical motor power D3 50Hz
Parallel motor power D4 50Hz
3 Medium speed transplant
3000 plants/h
Vibration D1 2560Hz
Transverse motor power D2 50Hz
Vertical motor power D3 50Hz
Parallel motor power D4 50Hz
4 High speed transplant
4500 plants/h
Vibration D1 2560Hz
Transverse motor power D2 50Hz
Vertical motor power D3 50Hz
Parallel motor power D4 50Hz

2.4. A hybrid evaluation method based on DT model and data-driven transplanting effect

Through the hybrid approach, the evaluation of the transplanting quality by data-driven methods is systematically observed, and the results are corrected by the empirical derivation theoretically. Figure 10 shows the steps of implementing the hybrid method based on DT model and data-driven approach: (1) the data-driven model is constructed and the monitoring of ETQ results is set as the observed values; (2)using a multi-physics field simulation, the DT model is converted into a state-space model for the hybrid algorithm settings; (3) a more accurate ETQ is calculated by the hybrid algorithm; (4) whether the ETQ has reached the threshold value is determined to make appropriate maintenance decisions or return to step (2) for iteration based on the judgment results.
The algorithm for intelligent recognition of healthy potted seedlings of vegetables mainly extracts and analyzes the image features of each potted seedling. A vegetable seedling’s leaf area M represents its growth conditions, so the leaf area features of vegetable seedlings are selected as one of the classification bases for the health recognition of potted seedlings. The digital image is composed of several square blocks of pixels, and the actual physical dimensions of the image pixels can be obtained by calibrating the camera size. Counting the number of pixels occupied by the mantle leaves in the image and converting the proportion can determine the actual area of the leaves. The threshold F is determined by the ratio of extracted leaf area to pore area in vegetable potted seedlings. The Heal algorithm for intelligent identification of healthy vegetable potted seedlings and rejection of inferior quality seedlings finally detects healthy vegetable potted seedlings by comparing the threshold value F.
Figure 10. DT-based and data-driven hybrid approach steps.
Figure 10. DT-based and data-driven hybrid approach steps.
Preprints 73037 g010
In fact, most of the obtained images of vegetable potting plants are disturbed by noise, and it is important to denoise the obtained images. The noise in image processing is mainly Gaussian noise. The three-dimensional block matching filtering (BM3D) algorithm is a denoising algorithm based on the three-dimensional transform domain, which is one of the best algorithms for processing video and image noise reduction. The algorithm is divided into two steps: First, a base valuation is obtained from the block matching 3D matrix transformation; then the noise image is filtered by the obtained base valuation, and the overlapping blocks are revalued and weighted average by the aggregation method to obtain the final image.
The basic principle of block matching is to divide an image into several parts of a specific size that do not overlap. Let the displacement of each pixel in each part be the same, a specific search area is selected, and then delineate it. The matching criterion is formulated to search for blocks that similar to the current block in the search area, that is, the matching block.
I represents the image containing noise; P represents any matching block that has been divided, and the block size of P is set to K × K. Q represents the sliding window block during the search process. If the block size is known, its upper left pixel point represents the matching block, P∈I, and Q∈I. During the block matching process, an appropriate step size h is determined first, and then the blocks are divided and searched based on the principle of top-to-bottom, left-to-right sequential approach. The current block P is selected as the reference block, and the area with P as the center point and the diameter d is taken as its search area,
Preprints 73037 i001
where S(P) is set as the three-dimensional matrices aggregated from similar blocks, and τd is the distance threshold during the search process. The distance d between the matching blocks in the search process is as follows:
Preprints 73037 i002
where: X is the matrix value of the matching block.
Finally, S(P) as the set of matrix blocks in the matrix is arranged in the order of d(P, Q) and a three-dimensional matrix TS(P) of K×K×S(P) is obtained. Then the denoising in the hree-dimensional transform domain can be expressed as follows:
Preprints 73037 i003
where N3D denotes the three-dimensional you-transformation of the three-dimensional matrix TS(P), and the operator is N3D. The equation of the function γ is as follows:
Preprints 73037 i004
where, λ3D is threshold parameter of hard threshold filtering, and σ is parameter of Gaussian white noise.
The advantage of this denoising method is that it can distinguish the noise from useful information in the strip image without any loss of energy. Useful information can be correctly distinguished from irrelevant information, such as noiTS(P)se. Because most of useful image information is at the top of the energy of the 3D matrix TS(P), and irrelevant information such as noise is often at the bottom of the 3D matrix. This feature allows the filtering by hard thresholds in the transform domain, which effectively removes noise while retaining most of useful image information. After filtering the noisy image, each block will have its corresponding estimate, and each pixel will have its corresponding estimate. NP denotes the non-zero values in the filtered matrix coefficients, and WP denoting the estimated values of the underlying weights of the current block is as follows:
Preprints 73037 i005
In equation (5), the base estimate of the 3D transform domain filtering is used to calculate the final estimated weights:
Preprints 73037 i006
As can be seen, the larger the estimated weights, the smaller the noise entrained. Calculate the average estimate of each overlapping block in order to obtain the final image.
The health of the potted seedlings is analyzed by using the threshold F and the unit leaf area of the Heal algorithm to classify the potted seedlings into healthy, sub-healthy, poor quality, and empty holes. The red "1" represents the healthy seedlings; the green "1" represents the sub-healthy seedlings; the yellow "1" represents the poor quality seedlings, and the blue “0 " represents empty holes. The Heal algorithm calibrates the seedling tray and outputs the information about the coordinate of each type of seedling in the virtual model of digital twin, as shown in Figure 11.
Figure 11. Virtual model mapping of digital twin.
Figure 11. Virtual model mapping of digital twin.
Preprints 73037 g011
A plant transplanting DT model with actual physical transplanter mapping capability was established in the previous paper. Through the mapping interface, the working conditions such as spindle speed, ambient temperature, seedling tray position and transplantation effect are transmitted to the DT model for model update. The updated DT model is simulated to calculate the operating state and effect of the transplanter. The collected data, such as vibration and image information, is used to train the machine learning model and determine whether the transplanting operation continues in another way, as shown in Figure 12.
Figure 12. Hybrid evaluation method of PFT transplanting effect.
Figure 12. Hybrid evaluation method of PFT transplanting effect.
Preprints 73037 g012

2.5. Development and configuration of the digital twin service system

Unity 3D, SolidWorks 2020 and 3 dsMax 2017 are used to develop the PFT digital twin system, as shown in Figure 13. SolidWorks 2020 is first used for the 3D modeling of the HNWC020 plant transplanting machine, and then the animation of the transplanting process is created in 3 dsMax 2017. Finally, Unity 3D is used to design the system interface and the interaction of the program. TCP/IP communication protocol is used to transfer the data from the physical machine to the digital twin system. Every 0.2 s, a piece of data from the operation of transplanting is obtained and stored in packets, and every 5 s, a packet is sent to the twin database. The digital twin system parses one packet every 5 s. The software runs at a frequency of 30 frames/s.
Figure 13. PFT Digital Twin Service System.
Figure 13. PFT Digital Twin Service System.
Preprints 73037 g013

3. Results and Discussions

3.1. Data-driven comparative analysis of plant factory transplanting equipment working conditions

To investigate the optimal working parameters of the plant transplanter, it is necessary to analyze the vibration characteristics under different working conditions. To study the acceleration state of the shifting manipulator of the plant transplanting equipment, three-axis acceleration sensors are placed on the beam above the transplanting manipulator. Figure 14 shows the vibration fitting cloud of the transplanter during horizontal sliding. The sampling frequency of the signal is 2,560 Hz, and the RMS value is counted every one second. The horizontal slider moves laterally for about 2 seconds during a single transplanting process.
Fast Fourier Transform (FFT) is used to extract the principal frequencies and harmonics of the signal. This feature describes the vibrational energy at a specific frequency. Three-dimensional vibrations are used to extract features. The FFT results indicate that these sensor data features have different frequencies.
Figure 14. Signal characteristics of X, Y, Z triaxial acceleration frequency domain.
Figure 14. Signal characteristics of X, Y, Z triaxial acceleration frequency domain.
Preprints 73037 g014aPreprints 73037 g014b
Through the FFT analysis of frequency domain, the mechanical vibration characteristics under different working conditions are visually reflected, as shown in Figure 14. As shown in Figure 15a), the vibration signal in the X,Y,Z-axis directions does not have obvious peaks in condition 1. The vibration amplitude in each axis direction is significantly increased in condition 2. In the low frequency range within 100Hz, the vibration is mainly in the X-axis direction in which two wave peaks appear. This may be due to the coupling resonance between the transplanting parts of the transplanter and the original test stand during the transplanting process. As shown in Figure 14c), the vibration amplitude in the low frequency range within 100 Hz increases significantly in the X-axis direction, with a maximum value of 2.45 m/s2. The vibration amplitude in the medium frequency range of 210-580 Hz increases significantly in the Z-axis direction, with a maximum value of 1.5 m/s2. As in Figure 14d), the amplitude in the X-, Y- and Z-axis directions increases significantly in condition 4. The large amplitude in the Z-axis direction will affect the transplanting accuracy and upright of potted seedlings, which means that it is not suitable for transplanting in condition 4. The above analysis reflects the vibration frequency domain characteristics of the transplanting machinery under different working conditions, but does not reflect the vibration energy of each time period.
From the above analysis, it can be seen that under conditions 1 and 2, the vibration amplitude is small, and the transplantation efficiency is low. Under condition4, it is easy to resonate in parts, easily causing damage to the transplanting parts. At the medium speed transplanting of 3000 plants/h, the high transplanting efficiency can be maintained, and the vibration signal of X, Y, Z-axis above the transplanting robot is relatively gentle, which is suitable for transplanting.

3.3. Comparative analysis of real-time image processing effects of transplanting potted seedlings

The comparative analysis of transplanting potted seedling classification algorithms under optimal working conditions can ensure that the image analysis system obtains higher quality image information to ensure the final recognition accuracy. As shown in Figure 17, four methods: optimal threshold method for threshold segmentation, traditional maximum inter-class variance method, optimal threshold method threshold segmentation combined with traditional Wiener algorithm, and optimal threshold method for threshold segmentation combined with 3D block matching filter algorithm, are selected for image information processing of potted seedlings.
To verify the rationality of the genetic algorithm based optimal threshold method in this article, the optimal threshold method based on genetic algorithm and the traditional maximum inter class variance method for threshold segmentation are performed on the original images of two pepper seedlings in Figure 15a), respectively. The threshold segmentation results are shown in Figure 15b) and Figure 15c). It can be seen that the image obtained by the optimal threshold segmentation based on genetic algorithm can clearly segment the leaves of pepper seedlings from the background. However, the image obtained by the traditional maximum interclass variance threshold segmentation fails to segment the leaves of pepper seedlings, and the image is blurred, not meeting the requirements. This comparison can verify the reasonableness of the optimal threshold method based on genetic algorithm for threshold segmentation. In addition, the following comparison of image denoising algorithms is only based on Figure 15 b). Figure 15 c) does not meet the requirements and cannot satisfy the conditions for further testing. To verify the rationality of the 3D block matching filtering algorithm selected in this paper, the two images in Figure 17b) are denoised based on the 3D block matching filtering algorithm and the conventional Wiener algorithm, respectively. The denoised results of the image are shown in Figure 15d) and Figure 15e).
Figure 15. Real-time image processing comparison of transplanting potted seedlings.
Figure 15. Real-time image processing comparison of transplanting potted seedlings.
Preprints 73037 g015
From the above figures, it can be seen that the traditional Wiener algorithm image denoising of Figure 15 b) and Figure 15 c) fails to remove the noise close to the leaves and larger areas, resulting in significant overall error. In contrast, using the three-dimensional block matching filtering algorithm of Figure 15 b) and Figure 15c) for image denoising, Figure 15 e), obtained by using the 3D block matching filtering algorithm for image denoising in Figure 17b) and 17c), can almost completely remove the noise generated by the background parts, except for the leaves of pepper seedlings, and the denoising effect is good. Through comparison, the rationality of the optimal threshold method for threshold segmentation combined with 3D block matching filtering algorithm for image denoising can be verified.

3.4. Evaluation of transplanting effects with digital twin virtual mapping reproduction

As shown in Figure 18, potted seedlings processed in real-time are discriminatively classified and mapped into a virtual seedling tray grid. The final recognition maps are shown in Figure 16a) and Figure 16b). The optimal threshold method is used for threshold segmentation; the 3D block matching filtering algorithm and traditional Wiener algorithm are used for image denoising, respectively. The red "1" represents healthy seedlings; the green "1" represents sub-healthy seedlings; the blue "0" represents poor quality seedlings, and the yellow number indicates the holes. The Heal algorithm calibrates the seedling tray and outputs the coordinates of each type of seedling, as shown in Figure 16.
Figure 16. Virtual mapping reproduction of digital twin seedling tray.
Figure 16. Virtual mapping reproduction of digital twin seedling tray.
Preprints 73037 g016
It can be seen that in Figure 16a), based on the 3D block matching filtering algorithm, the coordinate of inferior seedlings is (3, 1), and the coordinates of sub-healthy seedlings is (6, 3). Based on the traditional maximum interclass variance method, the coordinates of inferior seedlings are output as (3, 1). As shown in Figure 16 b), based on the 3D block-matching filtering algorithm, the coordinates of inferior seedlings are (3, 1), (5, 2), (7, 1), and that of sub-healthy seedlings are (1, 1), (2, 1), (4, 2), (5, 1), (6, 2). Based on the traditional maximum interclass variance method, the coordinates of inferior seedlings are (3, 1), (5, 2), and the coordinates of sub-healthy seedlings are (1, 1), (2, 1), (5, 1), (6, 2). Figure 17 reflects the accuracy of the digital twin virtual mapping of transplanted seedlings. From Figure 17, it can be seen that by combining the optimal threshold method and traditional Wiener algorithm for image threshold segmentation and denoising, the recognition rate of healthy seedlings can reach 94.3%. Through comprehensively using the optimal threshold method and 3D block matching filtering algorithm for image threshold segmentation and denoising, the recognition rate of healthy seedlings has reached over 96.10%. As such, this experiment is more suitable for using the optimal threshold method and 3D block matching filtering algorithm for image threshold segmentation and denoising, and the recognition accuracy is high.
Figure 17. Accuracy of digital twin virtual mapping for transplanted seedlings.
Figure 17. Accuracy of digital twin virtual mapping for transplanted seedlings.
Preprints 73037 g017
The digital twin model-based and data-driven transplanting effect evaluation method can monitor the working status and evaluate the transplanting effect of plant transplanters online. The method can overcome the poor adaptability and difficulty in updating physical models, greatly improving the monitoring and optimization efficiency of configuration parameters. At the same time, the generated virtual entity mapping is more intuitively reflected in the control interface, which can significantly reduce the dependence of equipment operators on relevant expertise. If VR/AR technology can be combined in the future, it will bring a better planting experience, and operators may also grow vegetables like video games.

4. Conclusions

This case focuses on evaluating the vibration and transplanting effect of the transplanting manipulator, which plays an intelligent decision-making role in the regulation of the transplanting system. The transplanting robot is an important part of PFT, and their condition directly affects the accuracy and quality of plant transplanting. It is not only the vibration of the transplanting manipulator that reduces the dimensional accuracy of the planting, but also the fatigue deformation of the machine, which shortens its service life. Traditionally, transplant monitoring is based on worker experience and statistics, which would lead to improper or excessive maintenance. A DT-based hybrid application is studied for real-time vibration monitoring and real-time evaluation of key components of transplanters. At the medium speed transplanting of 3000 plants/h, the high transplanting efficiency can be maintained, and the vibration signal of X, Y, Z-axis above the transplanting robot is relatively gentle, which is suitable for transplanting. In terms of transplanting effect evaluation, this study combines machine vision and artificial intelligence technology for real-time discrimination and classification of the video images after transplanting. The real-time evaluation of transplanting effect is obtained to provide a data support for intelligent decision-making of transplanting machine. Combined the optimal threshold method with the traditional Wiener algorithm, the identification rate of healthy potted seedlings can reach 94.3%. Through comprehensively using the optimal threshold method and 3D block matching filtering algorithm for image threshold segmentation and denoising, the recognition rate of healthy seedlings has reached over 96.10%. In the future, the implementation of cloud and edge-based DT model as well as the DT-based model migration learning will be further investigated.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

This work was Supported by the National Key Research and Development Program of China Sub-project (No. 2021YFD2000705).

Conflicts of Interest

The authors declare that they have no known competing financial interests or personal relationships that would appear to influence the work reported in this article.

References

  1. Ariesen-Verschuur, N., Verdouw, C., & Tekinerdogan, B. (2022). Digital Twins in greenhouse horticulture: A review. Computers and Electronics in Agriculture, 199 http://doi.org/10.1016/j.compag.2022.107183. [CrossRef]
  2. Fu, W., Gao, J., Zhao, C., Jiang, K., Zheng, W., & Tian, Y. (2022). Detection Method and Experimental Research of Leafy Vegetable Seedlings Transplanting Based on a Machine Vision. Agronomy-Basel, 12(11) http://doi.org/10.3390/agronomy12112899. [CrossRef]
  3. Ji, J. T., Chen, K. K., Jin, X., Pang, J., Chen, X., Wang, Z. Y., & Fan, J. Y. (2021). RESEARCH ON THE BEAM DAMAGE DIAGNOSIS METHOD FOR A HIGH-SPEED TRANSPLANTER BASED ON THE STRAIN MODE. Applied Engineering in Agriculture, 37(1), 25-32. http://doi.org/10.13031/aea.13186. [CrossRef]
  4. Ji, J., Chen, K., Jin, X., Wang, Z., Dai, B., Fan, J., & Lin, X. (2020). High-efficiency modal analysis and deformation prediction of rice transplanter based on effective independent method. Computers and Electronics in Agriculture, 168 http://doi.org/10.1016/j.compag.2019.105126. [CrossRef]
  5. Jin, X., Tang, L., Li, R., Zhao, B., Ji, J., & Ma, Y. (2022). Edge recognition and reduced transplantation loss of leafy vegetable seedlings with Intel RealsSense D415 depth camera. Computers and Electronics in Agriculture, 198 http://doi.org/10.1016/j.compag.2022.107030. [CrossRef]
  6. Jin, X., Yuan, Y., Ji, J., Zhao, K., Li, M., & Chen, K. (2020). Design and implementation of anti-leakage planting system for transplanting machine based on fuzzy information. Computers and Electronics in Agriculture, 169 http://doi.org/10.1016/j.compag.2019.105204. [CrossRef]
  7. Liu, Y., Mousavi, S., Pang, Z., Ni, Z., Karlsson, M., & Gong, S. (2022). Plant Factory: A New Playground of Industrial Communication and Computing. Sensors, 22(1) http://doi.org/10.3390/s22010147. [CrossRef]
  8. Luo, W., Hu, T., Ye, Y., Zhang, C., & Wei, Y. (2020). A hybrid predictive maintenance approach for CNC machine tool driven by Digital Twin. Robotics and Computer-Integrated Manufacturing, 65 http://doi.org/10.1016/j.rcim.2020.101974. [CrossRef]
  9. Pylianidis, C., Osinga, S., & Athanasiadis, I. N. (2021). Introducing digital twins to agriculture. Computers and Electronics in Agriculture, 184 http://doi.org/10.1016/j.compag.2020.105942. [CrossRef]
  10. Rahul, K., Raheman, H., & Paradkar, V. (2019). Design and development of a 5R 2DOF parallel robot arm for handling paper pot seedlings in a vegetable transplanter. Computers and Electronics in Agriculture, 166 http://doi.org/10.1016/j.compag.2019.105014. [CrossRef]
  11. Sun, J., Sun, W., Zhang, G., Lv, B., & Li, C. (2022). High efficient production of plant flavonoids by microbial cell factories: Challenges and opportunities. Metabolic Engineering, 70, 143-154. http://doi.org/10.1016/j.ymben.2022.01.011. [CrossRef]
  12. Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., & Sui, F. (2018). Digital twin-driven product design, manufacturing and service with big data. International Journal of Advanced Manufacturing Technology, 94(9-12), 3563-3576. http://doi.org/10.1007/s00170-017-0233-1. [CrossRef]
  13. Tao, F., Sui, F., Liu, A., Qi, Q., Zhang, M., Song, B., Guo, Z., Lu, S. C. Y., & Nee, A. Y. C. (2019). Digital twin-driven product design framework. International Journal of Production Research, 57(12), 3935-3953. http://doi.org/10.1080/00207543.2018.1443229. [CrossRef]
  14. Tao, F., Zhang, M., Cheng, J., & Qi, Q. (2017). Digital twin workshop:a new paradigm for future workshop. Computer Integrated Manufacturing Systems, 23(1), 1-9.
  15. Tong, J., Qiu, Z., Zhou, H., Bashir, M. K., Yu, G., Wu, C., & Du, X. (2022). Optimizing the path of seedling transplanting with multi-end effectors by using an improved greedy annealing algorithm. Computers and Electronics in Agriculture, 201 http://doi.org/10.1016/j.compag.2022.107276. [CrossRef]
  16. Van Henten, E. J. (2019). Automation and robotics in greenhouses. In L. Marcelis & E. Heuvelink (Eds.), (63, pp. 359-378). http://doi.org/10.19103/AS.2019.0052.14. [CrossRef]
  17. Verdouw, C., Tekinerdogan, B., Beulens, A., & Wolfert, S. (2021). Digital twins in smart farming. Agricultural Systems, 189 http://doi.org/10.1016/j.agsy.2020.103046. [CrossRef]
  18. Wang, X., Vladislav, Z., Viktor, O., Wu, Z., & Zhao, M. (2022). Online recognition and yield estimation of tomato in plant factory based on YOLOv3. Scientific Reports, 12(1) http://doi.org/10.1038/s41598-022-12732-1. [CrossRef]
  19. Wang, X., Wu, Z., Jia, M., Xu, T., Pan, C., Qi, X., & Zhao, M. (2023). Lightweight SM-YOLOv5 Tomato Fruit Detection Algorithm for Plant Factory. Sensors, 23(6) http://doi.org/10.3390/s23063336. [CrossRef]
  20. Wang, X., Zubko, V., Onychko, V., & Zhao, M. (2022). Development status and trend of plant factory intelligence in China. Scientific Bulletin. Series F. Biotechnologies, 26(1), 65-70.
  21. Yang, Q., Ahmad, I., Faheem, M., Siddique, B., Xu, H., & Addy, M. (2021). Development and assessment of belt-drive seedlings transmission device for fully-automatic vegetable transplanter. Computers and Electronics in Agriculture, 182 http://doi.org/10.1016/j.compag.2020.105958. [CrossRef]
  22. Zhang, L., Yang, X., Li, T., Gan, R., Wang, Z., Peng, J., Hu, J., Guo, J., Zhang, Y., Li, Q., & Yang, Q. (2022). Plant factory technology lights up urban horticulture in the post-coronavirus world. Horticulture Research, 9 http://doi.org/10.1093/hr/uhac018. [CrossRef]
  23. Zhang, P., & Li, D. (2022). EPSA-YOLO-V5s: A novel method for detecting the survival rate of rapeseed in a plant factory based on multiple guarantee mechanisms. Computers and Electronics in Agriculture, 193 http://doi.org/10.1016/j.compag.2022.106714. [CrossRef]
  24. Zhao, Y., Lian, L., Bi, C., & Xu, Z. (2022). Digital twin for rapid damage detection of a fixed net panel in the sea. Computers and Electronics in Agriculture, 200 http://doi.org/10.1016/j.compag.2022.107247. [CrossRef]
  25. Zhuang, C., Liu, J., Xiong, H., Ding, X., Liu, S., & Weng, G. (2017). Connotation,architecture and trends of product digital twin. Computer Integrated Manufacturing Systems, 23(4), 753-768.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

Subscribe

© 2024 MDPI (Basel, Switzerland) unless otherwise stated