In this paper, we propose a robust video understanding model for activity recognition by learning the actor’s pair-wise correlations and relational reasoning, exploiting spatial and temporal information. In order to measure the similarity between the pair appearances and construct an actor relations map, the Zero Mean Normalized Cross-Correlation (ZNCC) and the Zero Mean Sum of Absolute Differences(ZSAD) is proposed to allow the Graph Convolution Network (GCN) to learn how to distinguish group actions. We recommend that MNASNet be used as the backbone to retrieve features. Experiments show a 38.50% and 23.7% reduction in training time in the 2-stage training process along with a 1.52% improvement in accuracy against traditional methods.