The real systems are often complex, nonlinear, and noisy in various areas including mathematics, natural science, and social science. We present the symplectic entropy (SymEn) measure as well as an analysis method based on SymEn to estimate the nonlinearity of the complex system by analyzing the given time series. The SymEn estimation is a kind of entropy based on symplectic principal component analysis (SPCA) which represent organized but unpredictable behaviors of systems. The key to SPCA is to preserve the global submanifold geometrical properties of the systems through symplectic transform in the phase space, which is a kind of the measure-preserving transforms. The capability of preserving the global geometrical characteristics makes the SymEn a test statistic to detect the nonlinear characteristics in several typical chaotic time series and the stochastic characteristic in the Gaussian white noise. The results are in agreement with findings in the approximate entropy (ApEn), the sample entropy (SampEn) and the fuzzy entropy (FuzzyEn). Moreover, the SymEn method is also used to analyze the nonlinearities of the real signals (including the EEG signals for ASD and healthy subjects, and the sound and vibration signals for the mechanical systems). The results indicate that the SymEn estimation can be taken as a measure for describing the nonlinear characteristics in the data collected from the natural complex systems.