The traffic light detection and recognition are crucial for enhancing the security of unmanned systems. This study proposes a YOLOv5-based traffic light detection algorithm to tackle the challenges posed by small targets and complex urban backgrounds. Initially, the mosaic-9 method is employed to enhance the training dataset, thereby boosting the network’s ability to generalize and adapt to real-world scenarios. Furthermore, the network incorporates the Squeezed-and-Excitation (SE) attention mechanism to improve. Moreover, the YOLOv5 algorithm’s loss function has been optimized by substituting it with EIoU_loss, which addresses issues like missed detection and false alarms. Experimental results demonstrate that the model , trained with this enhanced network, achieves a 99.4% mAP on a custom dataset, which is 6.3% higher than the original YOLOv5, while maintaining a detection speed of 74 f/s. Therefore, this algorithm offers higher detection accuracy and effectively meets real-time operational requirements.