The emergence of unmanned aerial vehicles (UAVs) raised multiple concerns, given their potential for malicious misuse in unlawful acts Vision-based counter-UAV applications offer a reliable solution compared to acoustic and radio frequency-based solutions because of their high detection accuracy in diverse weather conditions. The existing solutions work well on trained datasets, but their accuracy is relatively low for real-time detection. In this paper, we model deep learning-empowered solutions to improve the multi-class UAV's classification performance using single-shot object detection algorithms YOLOv5 and YOLOv7. The transfer learning is employed for performance improvement and rapid training with improved results. We customized a multi-class dataset containing multi-rotor, fixed-wing, and single-rotor UAVs in challenging weather conditions. Experiments show that the integration of transfer learning has achieved good results, with an overall best average-classification precision of 94\%, an average recall of 93.1\%, a mAP$@$0.5 average of 95.3\%, and an average F1 score of 92.33\%. The dataset and code are available as an open source: https://github.com/ZeeshanKaleem/YOLOV5-Large-vs-YOLOV7.git