Improved YOLOv8-seg Based on Multi-scale Feature Fusion and Deformable Convolution for Weed Precision Segmentation

1. Introduction

The issue of weeds inevitably impacts crop yields of agricultural production. In China, there are 1,430 species (varieties) of agricultural weeds, leading to a reduction in grain production of up to 60 million tons and economic losses of up to 220 billion yuan. Current main weeding control methods include biological weeding, chemical weeding, traditional mechanical weeding and targeted weeding [1,2,3]. Biological weeding involves introducing foreign organisms to suppress weed growth, but these organisms can threaten local ecosystems [4]; Extensive application of herbicides plays a crucial role in controlling weeds and boosting agricultural efficiency, yet, this extensive application can induce resistance in weeds and significantly affect soil, biota, and human health [5]; Traditional mechanical weeding is effective pre-emergence but post-emergence inaccuracy can irreversibly damage crops; Targeted weeding technology, which focuses on the specific locations of weeds, has gradually become a focal point in the era of precision agriculture and Agriculture 4.0 [6].

Current targeted weeding technologies can be categorized into targeted herbicide spraying and laser-targeted burning. With the rapid advancement of computer vision, deep learning convolutional neural networks have been widely implemented in targeted herbicide spraying [7,8,9,10]. Yu et al. [11], Li et al. [12], Babu et al. [13] and Zhao et al. [14] have made optimizations to DeepLabv3, YOLOv7, Deep Residual Convolutional Neural Network (DRCN), and the YOLOv4 backbone network, achieving detection accuracies of 91.53%, 94.96%, 97.3%, and 98.52%, respectively. Through precise targeted spraying, these enhancements have effectively reduced the amount of herbicides used. However, the use of herbicides cannot be completely eliminated, which contradicts the principles of sustainable green agriculture [15]. Laser-targeted weeding, as a branch of targeted weeding, has become feasible with the maturation of compact laser devices and the enhanced transferability of computer vision models. This technology allows for the precise control of lasers via high-speed visual computing on devices with limited computational power [10,16,17,18]. Therefore, laser-targeted weeding methods are poised to become a more efficient and environmentally beneficial solution for weed management [19]. In earlier research Xiong et al. [20], a color differentiation algorithm was used to extract plants from the soil background, followed by size differentiation to distinguish between crops and weeds. This method proved to be inefficient and less effective in complex agricultural fields. Zhu et al. [21,22] designed a neural network-based blue laser weeding robot, incorporating a lightweight attention module into the YOLOx-Darknet architecture. Experimental tests showed a weed detection accuracy of 88.94% with a seedling damage rate of 4.53%. However, there is still significant room for improvement in the model's precision. Fatima et al. [23] compared the Yolov5 and SSD-ResNet50 networks using datasets of three crops and four weeds, with localization via bounding boxes. Tests demonstrated that the Yolov5-trained model achieved 88% detection accuracy and 27 FPS when transferred to the Nvidia Xavier AGX standalone device. However, such recognition accuracy and localization methods do not meet the requirements for laser weeding.

The application of laser weeding has been primarily limited to bounding box localization (detection) of weeds, using enhanced target detection algorithms like ResNet and the YOLO series to meet practical application needs. However, Rakhmatulin et al. [9] experimental results from laser weeding indicate that the optimal time for weed removal is when the weed stem diameter is less than 2mm during the 3-4 leaf stage, necessitating precise targeting of the weed stem and leaves with lasers for effective removal. This introduces new requirements for control processing: Laser weeding equipment requires an understanding of the complete plant shape to accurately locate the boundaries of weed stems and leaves, addressing the issue of bounding box localization in target detection algorithms which fails to control the laser for precise cutting. Thus, this paper first applies image segmentation technology to the field of laser weeding and propose an improved YOLOv8 instance segmentation algorithm based on bidirectional feature fusion and deformable convolution (BFFDC-YOLOv8-seg). This improved instance segmentation algorithm supports precise localization of weeds during the laser weeding process, effectively reducing crop damage rates. In order to conduct this study, the following tasks need to be completed:

• The dataset images are processed by adding Gaussian noise and adjusting the color space to enhance the model's generalization and robustness towards weed edge features in agricultural environments.
• By obtaining model scales compatible with laser weeding equipment and developing pre-trained weights more suited for agricultural settings, the accuracy and speed of model training are enhanced.
• During the feature fusion process, a Bidirectional Feature Pyramid Network (BiFPN) is introduced. BiFPN's weighted multi-scale feature fusion improves the network's focus on small targets, addressing the issue of inconspicuous features in complex backgrounds.
• DSConv is integrated to enhance the network's capability to segment irregular edges of plant stems and leaves, enabling accurate weed segmentation.

2. Materials and Methods

2.1. Image Collection and Dataset Construction

2.1.1. Data Collection and Annotation

The dataset in this paper consists of images captured on-site in Guiyang, Guizhou Province, China. It includes eleven types of weeds: Amaranthus blitum, Cirsium arvense, Senna tora, Portulaca oleracea, Digitaria sanguinalis, Ipomoea nil, Euphorbia heyneana, Cyperus rotundus, Mollugo stricta, Platostoma palustre and Eleusine indica, as shown in Figure 1. All images are in JPEG format with a resolution of $640\times 640$ pixels. Images were manually annotated using LabelMe to acquire data labels in JSON format, which were then converted into TXT files containing multiple coordinate points.

2.1.2. Dataset Augmentation and Construction

Deep learning models depend strongly on the image features within a dataset. Rich image data improves model accuracy and helps prevent overfitting. Yet, noise in the images and errors in annotations can decrease the accuracy of the model. To compensate for insufficient data and prevent network overfitting, this paper enhances the dataset by applying $90°$ clockwise and counterclockwise rotations, adding 0.1% Gaussian noise, and varying $\pm 25\%$ saturation and $\pm 15\%$ brightness. These methods expanded the dataset from 1,320 to 3,168 images, as shown in Table 1. The original dataset is divided into training, validation, and test sets in a 7:2:1 ratio.

2.2. Network Model Construction

2.2.1. Structure of the YOLOv8-seg Network

The YOLO (You Only Look Once) series of algorithmic frameworks stand out among various detection methods due to their rapid detection capabilities and high precision [24]. With continuous updates and iterations of the model framework, the YOLO series has become a popular real-time object detection model, extensively used in precision and automated agriculture for the detection and segmentation of crops, pests, and weeds.

In 2023, the Ultralytics team introduced the latest YOLOv8 object detection algorithm, evolving from YOLOv5, as a single-stage anchorless detection framework. The overall network structure is divided into four main components: Input, Backbone, Neck, and Head. The Input section, as the interface, is responsible for scaling input images to the dimensions required for training. It features modules such as mosaic data augmentation, adaptive anchor calculations, adaptive image scaling, and Mixup data enhancement. The Backbone is an enhancement over the YOLOv5 model, adopting ELAN's design principles by replacing the C3 structure with a more gradient-rich C2f structure, enhancing feature extraction through additional skip connections and split operations, and varying channel numbers across different model scales to maintain lightness while capturing more gradient flow. The Neck intensifies feature integration across dimensions, following the Feature Pyramid Network (FPN) [25] and Path Aggregation Network (PAN) [26] architectures, with convolution operations in the upsampling phases removed in layers 4-9 and 10-15 compared to YOLOv5. The Head section employs the current mainstream decoupled structure (Decoupled-Head), separating the classification and detection heads. It replaces the traditional anchor-based approach with an anchor-free method, as shown in Figure 2. The original Objectness branch is removed, leaving only the decoupled classification and regression branches. Moreover, the regression branch utilizes the integral form representation proposed in Distribution Focal Loss, allowing each independent branch to focus more on its respective feature information. YOLOv8 instance segmentation (YOLOv8-seg), an extension of the YOLOv8 model for instance segmentation, enhances the base target detection model by incorporating the YOLACT [27] network to achieve pixel-level instance segmentation. The model outputs masks, class labels, and confidence scores for each object located in the image.

This paper proposes optimizations to the YOLOv8-seg network, enhancing its segmentation accuracy and small object detection capabilities specifically on a dataset of weeds in vegetable fields. Additionally, these improvements adapt the network to function on standalone devices with limited memory and computational power.

2.2.2. Structure of the BFFDC-YOLOv8-seg Network

To enhance the accuracy of weed segmentation in complex, cluttered agricultural fields with overlapping plants, while ensuring model simplicity and real-time segmentation, this paper introduces the new BFFDC-YOLOv8-seg instance segmentation network. As shown in Figure 3, the network reconstructs the Concat module in the original Neck structure, replacing the existing FPN and PAN with BiFPN for innovative multi-scale feature fusion, effectively enhancing the network's ability to detect small targets. The introduction of DSConv to replace some of the $3\times 3$ convolutions in the original Backbone, integrating features extracted by $3\times 3$ convolutions with DSConv, increases the flexibility of the convolution kernels, thus improving the accuracy of segmentation for irregular edges of plant stems and leaves. The BFFDC-YOLOv8-seg network achieves precise segmentation of highly similar, cluttered, and overlapping weeds in complex field backgrounds, with features that ensure simplicity and real-time performance, making it feasible for operation on standalone devices.

• Appropriate weight documents and scales

Ultralytics officially provides five different scales of networks (N/S/M/L/X) and corresponding initial weight files on Github to cater to various application scenarios. These training weight files, essential for assisting training, enhance the accuracy and speed of training, containing model parameters such as weights and biases for each layer. However, the official weight files, trained on the COCO2017 dataset, lack the capability to perceive vegetable field environments. To better acclimate the model to vegetable fields and achieve improved training outcomes, this paper utilizes a public plant instance segmentation dataset with over five thousand images. By iterating 200 times on the original YOLOv8(N/S/M/L/X)-seg networks, this study obtains training weights adapted to vegetable field environments, serving as optimized training weights for the model.

Table 2 clearly shows that under the same training batches, the S model has significantly lower inference speed compared to the N model, with only a 1.3% improvement in accuracy and a substantial increase in model size. The M/L/X models, compared to the N model, show a notable increase in size and a significant decrease in inference speed, with a maximum of only 1.9% improvement in accuracy. With no significant gains in detection accuracy, the larger models require substantial storage resources and higher processing power, making them unsuitable for resource-limited laser weeding devices. Therefore, this paper chooses to optimize the N-scale model.

• Multi-scale feature fusion

The Concat module in the Neck section, which includes both FPN and PAN, plays a critical role in the fusion of image information. As depicted in Figure 4a and described by Equation 1, traditional FPN networks use input features $P_i^{in}=\left(P_3^{in}\cdots P_7^{in}\right)$ with 3-7 layers, where $P_i^{in}$ denotes the resolution level $in/2^i$ of the input image. Features are aggregated from top to bottom, and $resize$ usually involves up-sampling or down-sampling for resolution matching, while $Conv$ is typically used for feature processing. This results in feature fusion being limited by a unidirectional flow of information, which is ineffective in extracting features of small weed targets in agricultural settings with high similarity and indistinct color features. To enhance the detection capabilities of small targets in agricultural environments, this paper introduces a Bi-directional Feature Pyramid Network (BiFPN) [28].

$$\begin{gathered} P_7^{out}=Conv\left(P_7^{in}\right) \\ {P}_6^{out}=Conv\left(P_6^{in}+resize\left(P_7^{out}\right)\right) \\ \cdots \\ {P}_3^{out}=Conv(P_3^{in}+resize(P_4^{out}\left)\right) \end{gathered}$$

(1)

BiFPN represents an efficient bidirectional framework for cross-scale connections and fast normalization of feature fusion. From the network topology Figure 4b,c, it can be seen that BiFPN modifies the multi-scale connections within the PAN architecture: Initially, it removes network nodes that have a single input feature edge without fusion, creating a simplified bidirectional network; Subsequently, when the original input and output nodes are at the same level, an additional pathway is added between them to enable more feature fusion without significantly increasing computational costs; Lastly, unlike PAN which features only one top-down and one bottom-up pathway, each bidirectional (top-down and bottom-up) pathway is implemented as a feature network layer, repeated multiple times to achieve advanced feature fusion.

Different input features have varying resolutions, and compared to the high resolution of crops, the smaller resolution of weed inputs leads to a significant imbalance in the network's output contributions. Traditional methods treat all input features equally without distinction, which is not ideal in practical applications. We consider adding an extra weight to each input, allowing the network to learn the importance of each feature, thus preventing the network from overlooking the small-scale features of weeds. Based on this concept, BiFPN uses Fast Normalized Fusion as described in Equation 2, an efficient and stable weighted fusion mechanism that applies a $ReLu$ activation function after each $w_i$ to ensure $w_i\ge 0$ and $\epsilon =0.001$ are small, keeping each normalized weight between 0 and 1. Fast normalized fusion, similar in learning behavior and accuracy to Equation 3 (Softmax-based Fusion), omits the $Softmax$ operation, allowing BiFPN to run 30% faster on GPUs. To further enhance efficiency, we use depthwise separable convolutions for feature fusion and add batch normalization and activation after each convolution.

$$O={\sum }_i\frac{w_i}{\epsilon +{\sum }_j{w}_j}·I_i$$

(2)

$$O={\sum }_i\frac{e^{w_i}}{{\sum }_j{e}^{w_j}}·I_i$$

(3)

The original Concat module has been restructured, using BiFPN to replace the traditional FPN and PAN for a novel feature fusion approach, thereby assigning higher weights to small object features, enhancing the focus on small targets.

• Deformable convolution

In laser weeding operations, targeting the critical tissue parts of weeds with laser beams is essential for effective weed eradication; imprecise targeting can increase the accidental injury rate to crop seedlings. The original YOLOv8-seg network relies on the detection accuracy of bounding boxes within the Backbone, but square bounding boxes are not sensitive to the local information of irregular targets. Therefore, to enhance the network's perception of the irregular edges of weeds, Deformable Convolutions (DCN) [29] are considered for integration into the Backbone, allowing some $3\times 3$ convolutional kernels to adjust their shapes to fit the irregular structures of weeds, while maintaining the stability of the convolutional structure and reducing deviation. Given that Dynamic Snake Convolution (DSConv) [30] performs well in segmenting tubular structures, adapting to slender and twisted local structural features to enhance geometric structure perception, this paper introduces DSConv, constructing convolutional kernels with strong perception of irregular curves.

This paper aims to extract the irregular local features of weed stems and leaf edges, assuming standard 2D convolutional coordinates $K$, with the center coordinate as $K_i=\left(x_i,y_i\right)$. The original $3\times 3$ convolutional kernel is $K$ represented as:

$$K=\left\{\left(x-1,y-1\right),\left(x-1,y\right),\cdots ,\left(x+1,y+1\right)\right\}$$

(4)

By introducing deformation offsets $∆$, the convolutional kernel becomes more flexible, focusing on the irregular edges of tubular weed stems and leaves. Figure 5 linearizes the standard kernel in both axial directions, expanding it into a kernel of size 9. Taking the$X$ axial direction as an example, each grid position in $K$ is denoted as: $K_{i\pm c}=\left(x_{i\pm c},y_{i\pm c}\right)$, where $c=\mathrm{0,1},\mathrm{2,3},4$ represents the horizontal distance from the center grid. The selection of each grid position $K_{i\pm c}$ in kernel $K$ is a cumulative process. Starting from the central position $K_i$, the position away from the center grid depends on the position of the previous grid: $K_{i\pm 1}$ increases by an offset $∆=\left\{\delta |\delta \in \left[-\mathrm{1,1}\right]\right\}$ relative to $K_i$. Therefore, the offsets need to be accumulated to ensure that the kernel conforms to a linear structural form.

The change in the X-axis direction is:

$$K_{i\pm c}=\left\{\begin{array}{c}\left(x_{i\pm c},y_{i\pm c}\right)=\left(x_i+c,y_i+{\sum }_i^{i+c}∆y\right)\\ \left(x_{i\pm c},y_{i\pm c}\right)=\left(x_i-c,y_i+{\sum }_{i-c}^i∆y\right)\end{array}\right.$$

(5)

The change in the Y-axis direction is:

$$K_{j\pm c}=\left\{\begin{array}{c}\left(x_{j\pm c},y_{j\pm c}\right)=\left(x_j+{\sum }_j^{j+c}∆x,y_j+c\right)\\ \left(x_{j\pm c},y_{j\pm c}\right)=\left(x_j+{\sum }_{j-c}^j∆x,y_i-c\right)\end{array}\right.$$

(6)

Since the offset $∆$ is typically a decimal, while coordinates are usually in integer form, bilinear interpolation is employed, expressed as:

$$K={\sum }_{k^{\mathrm{\text{'}}}}B\left(K^{\mathrm{\text{'}}},K\right)·K^{\mathrm{\text{'}}}$$

(7)

Here, $K$ represents the decimal positions in Equations 5 and 6. $K^{\text{'}}$ enumerates all integer spatial positions. $B$ is a bilinear interpolation kernel, which can be decomposed into two one-dimensional kernels, that is:

$$B\left(K,K^{\mathrm{\text{'}}}\right)=b\left(K_x,K_x^{\mathrm{\text{'}}}\right)·b\left(K_y,K_y^{\mathrm{\text{'}}}\right)$$

(8)

As shown in Figure 6, the changes in the two-dimensional ($X$ and Y axes) setup enable the dynamic serpentine convolutional kernels described in this paper to cover a $9\times 9$ perceptual field during their deformation, better adapting to elongated tubular structures and enhancing the perception of critical features. The use of $3\times 3$ convolutional kernels to perform the functions of 9x9 kernels allows for greater flexibility in the model's kernels while keeping scale increases minimal.

2.3. Model Training and Outputs

This study’s model training was conducted on a Windows 10 operating system. The computer equipped with a 13^th Gen Intel Core i5-13600KF CPU @ 5.1GHz, 32GB of RAM, and an NVIDIA GeForce RTX 4060 8GB GPU. Under the PyTorch deep learning framework, a multi-class segmentation task neural network training model was constructed, with the main software versions as shown in Table 3. The training configuration included an input image size of $640\times 640$, batch size of 16, momentum of 0.937, initial learning rate of 0.0001 and 50 iterations.

The trained model was output in onnx format and tested on an embedded edge computing device (NVIDIA Jetson Orin nano 4GB) running Ubuntu 20.04 OS, featuring an arm64 CPU (512 CUDA cores) with TensorRT acceleration.

2.4. Model Evaluation Criteria

This paper evaluates the model’s detection accuracy and image segmentation capabilities through metrics such as Precision, Recall, and Mean Average Precision (mAP). Frames Per Second (FPS) measures the inference speed on hardware, i.e., the number of images processed per second by the device.

Precision and Recall are calculated using a confusion matrix, which includes True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). Precision is the proportion of actual targets in the total predicted targets by the network, representing the classification accuracy of the network, whereas Recall is the ratio of true targets correctly predicted by the network to the actual number of true targets. The corresponding formulas are:

$$precision=\frac{TruePositive}{TruePositive+FalsePositive}$$

(9)

$$recall=\frac{TruePositive}{TruePositive+FalseNegative}$$

(10)

Intersection over Union (IoU) represents the ratio of intersection to union in object detection, used to measure the degree of overlap between the model-generated boundaries and the original annotated boundaries. When IoU exceeds 0.5, it is considered that the object is detectable. If the true region is A and the annotated region is B, then:

$$IoU=\frac{A\cap B}{A\cup B}$$

(11)

The Average Precision (AP) for a single category is determined by ranking the model’s predictions based on their confidence scores and calculating the area under the Precision-Recall (PR) curve, as follows:

$$AP={\int }_0^{1}precision\left(recall\right)d\left(recall\right)$$

(12)

Mean Average Precision (mAP) indicates the average precision across multiple categories, with mAP50 representing the mAP at a 50% IoU threshold. mAP50-95 is a stricter evaluation metric, calculating the average of the AP at each IoU threshold from 50% to 95%, with increments of 0.05, allowing for a more accurate assessment of model performance at various IoU thresholds.

$$mAP50-95=\frac{{AP}_{IoU=0.5+}+{AP}_{IoU=0.55}+{AP}_{IoU=0.6}+\dots +{AP}_{IoU=0.95}}{n}$$

(13)

3. Results

3.1. Ablation Experiments and Model Training Details

3.1.1. Ablation Experiments

This paper constructs ablation comparative experiments to verify the efficacy of optimization techniques, ensuring there is no competition or conflict among various methods, and that resource utilization is optimal to avoid inconsistencies and confusion in the model training and decision-making process. Two distinct optimization approaches were individually integrated into the original YOLOv8-seg network for comparative analysis of YOLOv8-seg, BiFPN YOLOv8-seg, DSConv YOLOv8-seg, and BiDS-YOLOv8-seg networks using the publicly available coco128 dataset, trained 50 batches as shown in Table 4.

The Introduction of BiFPN enhanced”the ’odel's precision, recall, mAP50, and mAP50:95 to 91.4%, 83.6%, 88.9%, and 64.1% respectively, showing improvements over the original model by 1%, 2.5%, 1.8%, and 0.4%. This indicates that BiFPN effectively enhances the model’s ability to fuse features across different scales. The introduction of DSConv resulted in an 8% increase in precision and a 1.6% increase in mAP50 compared to the original model, indicating that deformable convolutions perform well in segmenting object edges, beneficial for the segmentation of weed stems and irregular leaf surfaces. However, recall and mAP50:95 are similar to the original network, suggesting that DSConv’s feature recognition across multiple categories is somewhat limited, leading to poor balance, and thus should not be used in isolation.

The mAP50 curves for different models across training batches are shown in Figure 7. The BiDS-YOLOv8-seg model, formed by the fusion of BiFPN, DSConv, and YOLOv8-seg, exhibits the best detection performance with significant improvements in all metrics, achieving a precision of 91.7%, recall rate of 83.5%, and mAP50 of 89.3%.

3.1.2. Training Results for the BFFDC-YOLOv8-seg

In this study, we trained the BFFDC-YOLOv8-seg on a vegetable field weed dataset, with Figure 8 detailing various performance metrics during the training and validation processes. During model training, we observed that the Box Loss (box_loss) rapidly decreased from a high initial value to below 0.2 and converged, demonstrating high precision in detecting weed boundaries without overfitting; Segmentation Loss (seg_loss) also showed a significant decreasing and converging trend, validating the model’s effectiveness in segmentation tasks; The decrease in Classification Loss (cls-loss) indicated a gradual enhancement in the model’s reliability for object classification; The reduction in the specific loss metric, Direction/Flow Loss (dfl_loss), reflected the model’s learning capability in predicting object direction or flow. During model validation, all loss metrics showed a converging trend similar to training, with final stable values slightly higher than training results, indicating that the model also has good generalization capabilities on data outside the training set. Moreover, the Mask’s Precision (precision(M)) and Recall (recall(M)) remained above 0.9, with the Mean Average Precision at 50% IoU (mAP50) exceeding 0.9, and at a stricter 50%-95% IoU (mAP50-95) also demonstrating performance above 0.8, further confirming the model’s strong segmentation performance and robustness under rigorous evaluation standards.

In summary, the final loss values are low and stable, a good balance is achieved between precision and recall, and the model performs excellently in segmenting small weed targets at different IoU thresholds, effectively learning weed characteristics without overfitting, achieving excellent training results.

3.1.3. BFFDC-YOLOv8-seg Detection and Segmentation Effect

This study tests the BFFDC-YOLOv8-seg model using reserved test set images to verify its actual detection and segmentation effects. A normalized confusion matrix for each weed category was created, as shown in Figure 9, and results were assessed based on the classification accuracy of the weeds. The matrix results indicate that the model performs uniformly in detection, achieving 100% accuracy for Amaranthus, Cirsium, Digitaria, Eleusine, Portulaca, 96% for Platostoma, Cyperus, Senna, Mollugo, 92% for Euphorbia, and 89% for Ipomoea. During testing, the model exhibited numerous errors in detecting Euphorbia and Ipomoea, missing some Euphorbia instances and misidentifying some Ipomoea as Cirsium or Senna. This issue could be mitigated by collecting more data to enhance the model's ability to distinguish similar physical features such as the shape, texture, and color of leaves.

By comparing masks, the study found that the original model (YOLOv8-seg) failed to adequately segment weed stems and edges and missed small targets, as shown in Figure 10. The analysis suggests that the original network's insufficient convolution flexibility and limited ability to perceive irregular edges resulted in the omission of stems and edges; inadequate feature fusion also led to the non-detection of small targets. The weighted bi-directional multi-scale feature fusion in BFFDC-YOLOv8-seg model enhances the network's ability to perceive small targets; DSConv, with its more flexible convolution kernels, has a stronger ability to detect stems and irregular edges, improving edge segmentation accuracy. Experiments demonstrate that in complex agricultural environments, our model outperforms the original in extracting small target features and segmenting irregular edges. Although our model exhibits superior segmentation capabilities for irregular small targets, there remains room for improvement in edge segmentation.

3.2. Comparison of the Performance with the Other Segmentation Models

To validate the efficiency and advantages of the BFFDC-YOLOv8-seg weed segmentation model in this study, we compared it with mainstream segmentation networks such as Mask-RCNN [31] , YOLOv5-seg [32] , and YOLOv7-seg [33]. The same dataset and training parameters were used for training the models, and tests were conducted on the test set. The complete experiment assessed the accuracy (P), recall (R), mean average precision (mAP), inference speed (FPS), and model size for each segmentation network, as shown in Table 5.

From a comprehensive comparison of tests, it is evident that the BFFDC-YOLOv8-seg model proposed in this paper exceeds the original YOLOv8-seg model by 2.8% in mAP50 for segmenting small weed targets, and surpasses the Mask-RCNN, YOLOv5-seg, and YOLOv7-seg models by 10.8%, 13.4%, and 1.3% respectively, showing the highest precision in detecting and segmenting weeds in complex agricultural environments. In terms of real-time detection capabilities, Mask-RCNN with its complex model and high-density computations and YOLOv7-seg’s large-scale network, require high computational costs, resulting in inference speeds of only 34 and 18.3 FPS respectively. After increasing the sampling scale, this paper's network has a reduced inference speed compared to YOLOv8-seg's 270 FPS, but at 101 FPS, it fully meets the real-time detection needs of slow-moving laser weeding device, while maintaining optimal segmentation accuracy. Regarding model size, the network retains its lightweight characteristics, making the 6.8MB model more suitable for deployment on resource-limited automatic laser weeders compared to the Mask-RCNN and YOLOv7-seg models.

3.3. Testing on Standalone Devices

To ensure that the BFFDC-YOLOv8-seg model adapts well to standalone devices, this paper ports the trained model to the standalone embedded device Jetson Orin nano (4GB) for compatibility testing. The Nvidia Jetson, as an embedded AI computing platform, significantly reduces the computational cost of deep learning models, facilitating broader application of compact laser weeding devices. This paper also uses precision, recall, mAP50, mAP59-95, and FPS metrics to evaluate the model's performance on standalone devices. As shown in Table 6, the model achieves a 95.8% mAP50 on the Jetson Orin nano and processes images at 24.8 FPS, suitable for real-time weed segmentation.

4. Discussion

In research on automated laser weeding, many scholars focus on object detection algorithms, neglecting the coarseness of bounding box localization, which can lead to the accidental damage to seedlings and incomplete eradication of weeds. Conversely, with significant improvements in the efficiency and portability of segmentation algorithms, instance segmentation techniques are more advantageous for locating weeds in complex environments. Yue et al. [34] reported an mAP50 of 92.2% using an improved YOLOv8-seg model for segmenting tomato diseases at different stages, demonstrating significant potential of the YOLOv8 series in segmentation tasks.

Feature extraction of small and irregular objects has always been a hot topic in computer vision, with multi-scale feature fusion and dynamic deformable convolutions being the most common in addressing segmentation of such objects. In agricultural production, there are many applications for the segmentation of small targets, such as determining the extent of infection in plants by segmenting and locating plant pests and diseases. Therefore, precise segmentation of irregular small targets in complex backgrounds remains a significant challenge in agricultural applications.

This study demonstrates that the BFFDC-YOLOv8-seg effectively enhances weed segmentation capabilities in complex agricultural environments, making it suitable for small, cost-effective, automated laser weeding devices and capable of efficiently detecting and segmenting weeds in vegetable fields. Although this research has made some progress, many key technical issues still need to be addressed to make laser-targeted weeding equipment more aligned with practical needs: a more comprehensive dataset of field weeds needs to be established to increase the diversity of feature collection and further enhance the model's accuracy and generalizability; the model's detection precision needs further optimization and its complexity reduced to meet the real-time demands and resource constraints of practical applications; image coordinates obtained from the model's segmentation need to be integrated with a positioning algorithm for precise target localization; a mismatch between the speed of the laser source and detection efficiency leads to reduced efficiency, necessitating a more efficient laser source.

5. Conclusions

Addressing the challenges of inaccurate weed targeting and high crop damage rates associated with traditional laser weeding equipment. This study is the first to apply image segmentation technology to the field of laser weeding, using instance segmentation to precisely guide the laser targeting. This paper proposes a BFFDC-YOLOv8-seg weed segmentation model that integrates dynamic snake convolution and a weighted bidirectional feature pyramid network to precisely locate the boundaries of weed stems and leaves, addressing the issues of inaccurate weed targeting and high crop damage rates in traditional laser weeding equipment. In the original YOLOv8 network, weighted multi-scale feature fusion is implemented to enhance the network's ability to perceive small objects; DSConv replace some $3\times 3$ convolutional kernels, making the existing convolutional layers more flexible and better adapted to the irregular edges of weed stems and leaves. Experimental results demonstrate that the BFFDC-YOLOv8-seg model achieves an accuracy of 97.5%, a recall of 97.5%, an mAP50 of 98.8%, and an mAP50-95 of 84.2% on the test set. Compared to the current mainstream models such as Mask-RCNN, YOLOv5-seg, YOLOv7-seg, and YOLOv8-seg, this model achieves better mAP50 values, with improvements of 10.8%, 13.4%, 1.3%, and 2.8%, respectively. Additionally, the model size is only 6.8 MB, and in stability tests on the Jetson Orin nano independent device, the actual scenes captured by the camera can be processed at a detection speed of 24.8 FPS with an mAP50 of 95.8%. The proposed model in this paper delivers precise real-time weed segmentation, making it suitable for small, cost-effective automatic laser weeding devices. Although this research has achieved certain results, further optimization of the model's detection precision and reduction in complexity remains a focus. We hope that our efforts will lead to widespread adoption of laser weeding devices equipped with the BFFDC-YOLOv8-seg model in agricultural production, promoting sustainable green agriculture.