Advances in early detection of Breast cancer and treatment improvements have significant-ly increased survival rates. Traditional screening methods, including mammography, MRI, ultrasound, and biopsies, while effective, often come with high costs and risks. Recently, thermal imaging has gained attention due to its minimal risks compared to mammography, although it is not widely adopted as a primary detection tool since it depends on identifying skin temperature changes and lesions. The advent of machine learning (ML) and deep learning (DL) has enhanced the effectiveness of breast cancer detection and diagnosis using this technology. In this study a novel methodology for developing an interpretable computer-aided diagnosis (CAD) system for breast cancer detection, leveraging explaina-ble Artificial Intelligence (XAI) throughout its various phases. To achieve these goals, we proposed a new multi-objective optimization approach named Hybrid Particle Swarm Optimization algorithm (HPSO) and Hybrid spider Monkey Optimization algorithm (HSMO). These algorithms simultaneously combine the continuous and binary representations of PSO and SMO to effectively manage trade-offs between Accuracy, feature selection and hyperparameter tuning. We evaluate several CAD models and investigate the impact of handcrafted methods such as Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG), Gabor filters, and edge detection. We further shedding light on the effect of feature selection and optimization on feature attribution and model decision-making processes using the SHapley Additive exPlanations (SHAP) framework, with a particular emphasis on cancer classification using the DMR-IR dataset. The results of our experiments demonstrate in all trials that the performance of the model is improved. Also with HSMO our models achieved a high accuracy of 98.27% and F1- score of 98.15% while selecting only 25.78% of the HOG features. This approach not only boosts the performance of CAD models but also ensures comprehensive interpretability. This method emerges as a promising and transparent tool for early breast cancer diagnosis.