Evidence regarding the effect of psychological factors and evacuation on cardiovascular disease occurrence after large-scale disasters is limited. This prospective study followed up a total of 37,810 Japanese men and women aged 30–89 years from the Fukushima Prefecture with no history of stroke or heart disease at baseline (2012), until 2017. This period included 3000 cardiovascular events recorded through questionnaires and death certificates. The participants’ psychological distress, trauma reaction, and evacuation status were defined, and divided into four groups based on combinations of psychological factors and evacuation status. We calculated the hazard ratios and 95% confidence intervals for only psychological, only evacuation, or both of them compared with neither using Cox proportional hazard models. Psychological factors along with evacuation resulted in approximately 5% to 25% higher magnitude of stroke and heart disease risk than psychological factors only among men. Compared to neither, the multivariable hazard ratios of those with both psychological distress and evacuation were 1.75 for stroke and 1.49 for heart disease, and those of both trauma reaction and evacuation were 2.01 and 1.57, respectively, among men. Evacuation combined with psychological factors increased the risk of stroke and heart disease risks especially in men after the Great East Japan Earthquake.