Pedestrian detection is the core of driver assistance system, which collects the road conditions through the radars or cameras on the vehicle, judges whether there is a pedestrian in front of the vehicle, supports decisions such as raising the alarm, automatically slowing down or emergency stopping to keep pedestrians safe, and improves the security when the vehicle is moving. Suffered from weather, lighting, clothing, large pose variations and occlusion, the current pedestrian detection still has a certain distance from the practical applications. In recent years, deep networks have shown excellent performance for image detection, recognition and classification. Some researchers employed deep network for pedestrian detection and achieve great progress, but deep networks need huge computational resources which make it difficult to put into practical applications. In real scenarios of autonomous vehicle, the computation ability is limited. Thus, the shallow networks such as UDN (Unified Deep Networks) is a better choice since it performs well on consuming less computation resources. Base on UDN, this paper proposes a new deep network model named as two-stream UDN, which augments another branch for solving traditional UDN’s indistinction of the difference between trees / telegraph poles and pedestrians. The new branch accepts the upper third part of the pedestrian image as input, and the partial image has less deformation, stable features and more distinguished characters from other objects. For the proposed two-stream UDN, multi-input features including HOG feature, Sobel feature, color feature and foreground regions extracted by GrabCut segmentation algorithms are fed. Compared with the original input of UDN, the multi-input features are more conducive for pedestrian detection since the fused HOG features and significant objects are more significant for pedestrian detection. Two-stream UDN is trained through two steps: First, the two sub-networks are trained until converge; then we fuse results of the two subnets as the final result and feed it back to the two subnets to fine tune network parameters synchronously. To improve the performance, Softplus is adopted as activation function to obtain faster training speed, and positive samples are mirrored and rotated with small angle to make positive and negative samples more balanced.