Human epidermal growth factor receptor 2 (HER2 ) is an established gene associated with more aggressive and decreased survival of breast cancer (BC). HER2 grading is standard procedure for BC patients to determine their suitability for giving Trastuzumab, an US Food and Drug Administration (FDA) approved therapy for the treatment of HER2 positive BC patients. A HER2 positive patient benefits from the therapy but it may cause cardiac toxicity if given to a negative patient. Therefore, selecting the right patients to give this treatment is crucial. Fluorescence in-situ hybridization (FISH) and chromogenic in-situ hybridization (CISH) are the FDA-approved tests for HER2 grading. Clinically, the assessment is performed by counting signals manually from FISH slides. Existing automated methods fail when the color and shape of the nuclei and biomarkers vary. Moreover, a method designed for CISH doesn’t withstand FISH and vice-versa. In this study, we propose a robust HER2 grading system utilizing the segment anything model (SAM). The proposed system works for both FISH and CISH images regardless of the color and shape variations. We determined the HER2 status using the proposed system from serial-sectioned CISH and FISH specimens for the same patient, separately. Then, the results were compared with the pathologist’s manual FISH counts, which is the clinical reference. The proposed system achieved a high correlation using both FISH and CISH. The p-value (0.37) from the one-sided paired t-test and low bias (0.06) in the Bland-Altman plot ensured the system’s robustness for CISH and FISH based HER2 grading.