The practice of physical activity is an effective non-pharmacological strategy for preventing and treating chronic diseases. A large body of evidence has contributed to establishing a positive correlation between a physically active lifestyle and health benefits, including enhanced vaccination responses, lower numbers of senescent T-cells, increased T-cell proliferative capacity, lower levels of inflammatory cytokines, and improved neutrophil and macrophage function. While females are generally thought to exert more robust immune responses than males in response to a variety of challenges, and both male and female sex hormones have been suggested as mediators of immune function, research on this topic has not always been designed with a sex-specific lens. The goal of this review is to summarize the available experimental and clinical evidence linking exercise and immune function in male and female subjects, with an emphasis on sex differences and sex-specific mechanisms. Overall, the available evidence indicates that moderate exercise and physical activity improves immune function in both sexes, whereas prolonged and high-intensity exercise temporarily impairs immune responses at a higher degree in females. A role of male and female sex hormones in these sex-specific effects is also discussed.