Premature ventricular contraction (PVC) is one of the common ventricular arrhythmias, which may cause stroke or sudden cardiac death. Automatic long-term electrocardiogram (ECG) analysis algorithms could provide diagnosis suggestion even early warning for physicians, however, they are mutually-exclusive in terms of robustness, generalization and low complexity. In this study, a novel PVC recognition algorithm that combines deep learning-based heartbeat template clusterer and expert system-based heartbeat classifier is proposed. Long short-term memory-based auto-encoder (LSTM-AE) network was used to extract features from ECG heartbeats for K-means clustering. Thus, the templates were constructed and determined based on clustering results. Finally, the PVC heartbeats were recognized based on a combination of multiple rules, including template matching and rhythm characteristics. Three quantitative parameters, sensitivity (Se), positive predictive value (P+) and accuracy (ACC), were used to evaluate the performances of the proposed method on the MIT-BIH Arrhythmia database and the St. Petersburg Institute of Cardiological Technics database. Se on the two test databases was 87.51% and 87.92%, respectively; P+ was 92.47% and 93.18%, respectively; and ACC was 98.63% and 97.89%, respectively. The PVC scores on the 3rd China Physiological Signal Challenge 2020 training set and hidden test set were 36,256 and 46,706, respectively, which could rank first in the open-source codes. The results showed that the combination strategy of expert system and deep learning can provide new insights for robust and generalized PVC identification from long-term single-lead ECG recordings.