Infrared small target detection (ISTD) plays a crucial role in both civilian and military applications. The detection of small targets against dense cluttered backgrounds remains a challenging task, requiring the collaboration of false alarm source elimination and target detection. Existing approaches primarily focus on modeling targets while often neglecting false alarm sources. To address this limitation, we propose a Target and False Alarm Collaborative Detection Network to leverage the information provided by false alarm sources and the background. Firstly, we introduce a False Alarm Source Estimation Block (FEB) that estimates potential interferences present in the background by extracting features on multiple scales and employing stepwise upsampling for feature fusion. Subsequently, we propose a framework that employs multiple FEBs to eliminate false alarm sources across multiple scales. Finally, a Target Segmentation Block (TSB) is introduced to accurately segment the targets to produce the final detection result. Experiments on public datasets demonstrate that, compared to other methods, our model achieves high accuracy in segmenting targets while being able to extract false alarm sources which can be used for further studies.