This paper presents a robust method based on graph topology to find the topologically correct and consistent subset of inter-robot relative pose measurements for multi-robot map fusion. However, the absence of good prior gives a severe challenge to distinguish the inliers and outliers, and wrong loop closures can seriously corrupt the fused global map. Existing works mainly rely on the consistency of spatial dimension to select inter-robot measurements, which does not always hold. In this paper, we propose a fast inter-robot loop closure selection method that integrates the consistency and topology relationship of measurements, which both conform to the continuity characteristics of similar scenes and spatiotemporal consistency. The traditional high-dimensional consistency matrix is decomposed into the sub-matrix blocks corresponding to the overlapping trajectory area. Building on this logic, a clustering method involving topology correctness of inter-robot loop closures is introduced to split the entire measurement set into multiple clusters. We define the weight function to find the maximum cardinality subset with topologically correct and consistent, then convert the weight function to a maximum clique problem in the graph and solve it. We evaluate the performance of our method in a simulation and in a real-world experiment. Compared to state-of-the-art methods, the results show that our method can achieve competitive performance in accuracy while reducing computation time by 75%.