In this study, we identified the factors that affect the frequency of receiving nonmarket fruit and vegetables (FV). For Survey 1, we conducted a cross-sectional survey using a self-administered questionnaire for men aged 50–74 living in city (A) in Gunma, Japan. Participants were asked questions on FV receiving frequency, FV gardening, social cohesion (4–20 points), and basic characteristics. For Survey 2, a similar survey was conducted for residents aged 20–74 in three areas in city (B) in Gunma, but we included more variables. Ordinal logistic regression models were used for the analysis. In Survey 1, the responses of 243 participants were analyzed. The results showed that the FV receiving frequency was positively associated with non-gardeners and social cohesion. In Survey 2, the responses of 791 participants were analyzed. Vegetable receiving frequency was positively associated with rural and suburban areas, family structure, employment status, and non-farmers. The relationship between receiving frequency and social cohesion was similar to that found in Survey 1. In conclusion, in areas with flourishing FV cultivation, it seems to be easy to obtain FV through the social networks of reception, even for individuals who are not cultivating FV themselves.