The growing use of drones in precision agriculture highlights the need for enhanced operational efficiency. Despite the ability of computer vision based on deep learning has made remarkable progress in the past ten years, when it comes to segmentation task on UAVs, there is always a conflict between the demand of high precision and low inference latency. Due to such a dilemma, we propose the IRB-YOLO, an efficiency model based on Inverted Residual Block, devoting to provide constructive strategies in real-time detection tasks of UAV camera. The working details of this paper are as follows: (1) This paper innovates with a IR-Block(Inverted Residual Block), integrated into a refined YOLOv8-seg structure to create IRB-YOLO. This model specializes in pixel-level classification of UAV-acquired RGB images, facilitating the creation of exact maps to guide agricultural strategies. (2)When it comes to the experiments on a Vatica dataset with any other light-weight segmentation model, IRB-YOLO achieve at least a 3.3% increase in mAP. Further validation using a diverse species dataset confirms its robust generalization. (3)Without overloading the complex attention mechanism and deeper and deeper network, a stem that incorporates efficient feature extraction components, inverted residual block, can still possess outstanding modeling capabilities. IRB-YOLO builds a bridge between academic research and edge deployment of drones, making it applicable in real-world scenarios.