Diabetes is a risk factor for mild cognitive impairment (MCI) and dementia. However, how the clinical characteristics of type 2 diabetic patients with MCI are linked to sarcopenia and/or its criterion remain to be elucidated. Japanese patients with type 2 diabetes mellitus were categorized into the MCI group for MoCA-J (the Japanese version of the Montreal cognitive assessment) score <26, and into the non-MCI group for MoCA-J ≥26. Sarcopenia was defined by a low skeletal mass index along with low muscle strength (handgrip strength) or low physical performance (walking speed <1.0 m/s). Univariate and multivariate-adjusted odds ratio models were used to determine the independent contributors for MoCA-J <26. Among 438 participants, 221 (50.5%) and 217 (49.5%) comprised the non-MCI and MCI groups, respectively. In the MCI group, age (61 ± 12 vs. 71 ± 10 years, p < 0.01) and duration of diabetes (14 ± 9 vs. 17 ± 9 years, p < 0.01) were higher than those in the non-MCI group. Patients in the MCI group exhibited lower hand grip strength, walking speed, and skeletal mass index, but higher prevalence of sarcopenia. Only walking speed (rather than muscle loss or muscle weakness) was found to be an independent determinant of MCI after adjusting for multiple factors, such as age, gender, BMI, duration of diabetes, hypertension, dyslipidemia, smoking, drinking, eGFR, HbA1c, and history of coronary heart diseases and stroke. In subgroup analysis, a group consisting of male patients aged ≥65 years, with BMI <25, showed a significant OR for walking speed. This is the first study to show that slow walking speed is a sole determinant for the presence of MCI in patients with type 2 diabetes. It was suggested that walking speed is an important factor in the prediction and prevention of MCI development in patients with diabetes mellitus.