Directional protective relaying based on amplitude comparison of traveling wave on continuous co-phase auto-transformer(AT) power transmission system was proposed. Phase-model transformation is used for decoupling transient fault signals and get aerial mode component. The forward and backward traveling wave are obtained by calculating aerial mode component, then the intrinsic mode function(IMF) components can be obtained through Variational Mode Decomposition(VMD), the module maxima are obtained through calculating the IMF components. The fault direction is determined by the ratio of fault components. If two faults relays at the ends of electric traction networks detect a fault to be in the forward direction, the fault occurred in the internal area, the protection would operate properly. The simulation tests indicate that the protection scheme is feasible, and the proposed protection method can discriminate internal faults from external faults under various fault types, and its performances are immune to fault initial angle, ground resistance, etc.