Video behavior recognition often needs to focus on object motion processes. In this work, a self-organizing computational system oriented to behavioral clustering recognition is proposed, which achieves the extraction of motion change patterns by binary encoding and completes motion pattern summarization using a similarity comparison algorithm. And in the face of unknown behavioral video data, a self-organizing structure with layer-by-layer accuracy progression is used to achieve motion law summarization by using a multi-layer agent design approach. Finally, the real-time feasibility is verified in the prototype system using real scenes to provide a new feasible solution for unsupervised behavior recognition and space-time scenes.