The vibration signal of heavy gearbox presents non-stationary and nonlinear characteristics, which increases the difficulty to extract the fault feature. When the gear has a subtle fault, it may cause a perceptible change of local fluctuation rather than the large scale fluctuation. Therefore, the feature parameters extracted from local fluctuation can effectively improve the recognition performance of the gear fault. In this paper, a novel signal processing method based on variational mode decomposition (VMD) and detrended fluctuation analysis (DFA) is proposed to identify the gear fault of heavy gearbox. Firstly, the raw vibration signal is decomposed several mode components by VMD, which is an adaptive and non-recursive signal decomposition method. Next, the sensitive mode component is selected by a maximal indicator, which is composed of kurtosis and correlation coefficient of relative higher frequency mode components corresponding to local fluctuation of raw vibration signal. Finally, the characteristics of the double-scales feature parameters of selected sensitive mode are extracted by DFA. In addition, the position of turning point of double scales is estimated by sliding windowing algorithm. The proposed method is evaluated through its application to gear fault classification using vibration signal. The results demonstrates that the recognization rate of gear faults condition have marked improvement by proposed method than the DFA of Small Time Scale (STS-DFA) method.