The rampant misuse of drones poses a serious threat to national security and human life. Currently, the CNN method has been widely used in drone detection. However, there is a challenge that traditional CNN cannot cope with, which is small drone targets often have reduced amplitude or even lost features in infrared images. This paper proposes a Progressive Feature Fusion Network (PFFNet) that gradually increases the response amplitude of the target in the deep network. The Feature Selection Model (FSM) is designed to improve the utilisation of the output coding graph and enhance the feature representation of the target in the network. A lightweight segmentation head is also designed to achieve progressive feature fusion with multi-layer outputs. Experimental results show that the proposed algorithm can achieve low duration and high accuracy in drone target detection. On the public dataset, the IoU is improved by 2.53% and the detection time is reduced by 81.03%.