This study aims to develop an elderly care system for improving the interpersonal relationship of the elderly with mild cognitive impairment (MCI) by employing the speaker recognition technique and association functionality of social network platforms. Firstly, the speaker recognition units based on the Gaussian Mixture Model (GMM) and Gaussian Mixture Model-Universal Background Model (GMM-UBM) are implemented to identify the visitor via individual input utterance. After the visitor is identified, the proposed system will be linked to the private database and social network platforms to extract the associated message of two parties. Experimental results indicate that the speaker recognition unit based on GMM-UBM achieves the best performance. Finally, five elderly persons are invited to measure the usability of the proposed system. A questionnaire is used to survey the five elderly persons, and the result indicates that the proposed system is highly potentially applicable in improving the interpersonal relationship of the elderly with MCI.