Affective Computing Using an Emotion-Perception System for Interactive Experiences with Multi-Sensing Interfaces

Preprint

Article

Affective Computing Using an Emotion-Perception System for Interactive Experiences with Multi-Sensing Interfaces

Altmetrics

Downloads

166

Views

Comments

This version is not peer-reviewed

Submitted:

28 August 2024

Posted:

29 August 2024

You are already at the latest version

Alerts

Abstract

Affective computing has emerged as a prominent technology in recent years, driven by the increasing demand for machines capable of understanding and interpreting emotions. The COVID-19 pandemic has intensified this need by reducing physical interactions and increasing reliance on digital communication, which often involves complex emotional exchanges. To enhance social interactions by identifying and addressing negative emotions, an interactive emotion-perception system based on multi-sensing interfaces and affective computing has been proposed. Integration of gesture, voice, and brainwave sensors is employed to provide users with a rich sensory experience and immersive environment. Design principles and system development processes are derived from the affinity diagram method and prototype development approach. The proposed system was constructed, and experiments were conducted with 60 participants. Interpersonal interaction scenarios were explored to understand emotional transmission mechanisms, aiming to enhance user satisfaction and positive interpersonal experiences. Data were collected through indirect observation and questionnaires, followed by evaluation and statistical analysis. Findings include: (1) The emotion-perception system enhances positive interpersonal interactions by identifying 36 distinct emotions, with overall user emotions being positive but exhibiting notable individual variations; (2) The system effectively stimulates positive emotions and strengthens interpersonal interactions, with affective computing technology fostering enjoyable emotional resonance and high-quality relationships; (3) A positive correlation was observed between semantic analysis and brainwave metrics of relaxation and concentration, showing increased concentration and enhanced positive emotional interactions during message transitions; and (4) The multi-sensing and affective computing technology provides a pleasant and relaxing emotional journey, with interactions shifting from enthusiasm to comfort, demonstrating the system’s capacity for a satisfying and enjoyable experience.

Keywords:

Subject: Computer Science and Mathematics - Other

1. Introduction

1.1. Background

The COVID-19 pandemic has led to widespread mental health issues, including depression, anxiety, and loneliness. Dubé et al. [1] found a strong link between pandemic-induced social problems and negative emotional behaviors such as depression. As the pandemic eased, social media became a common way to seek comfort and connection, offering low-cost, flexible, and unrestricted interaction, thus significantly enhancing interpersonal connections [2]. The World Health Organization (WHO) has designated October 10th each year as “World Mental Health Day” to emphasize the importance of mental health in shaping our thoughts, feelings, and behaviors. The day encourages people globally to foster positive, relaxed, and constructive emotions [3].

As AI evolves, high-efficiency computing alone no longer meets user needs. Users increasingly seek machines that can understand emotions, driving demand for emotional sensing technology. Affective computing, an interdisciplinary field involving physiology, sociology, cognitive science, mathematics, psychology, linguistics, and computer science, holds significant research value. It uses various electronic materials to sense and collect emotional data from voices, facial expressions, and behaviors, creating systems for emotion recognition [4].

Collins [5] proposes in his interaction ritual chains theory that rituals are normative behaviors that foster mutual attention and emotional resonance through rhythmic interactions. This theory demonstrates that both the content and context of interactions influence the overall experience. When combined with sociological theories, it reveals that interactions involve emotions, which subsequently generate a series of rituals [6].

Users’ cognitive psychology influences their interaction behaviors, with emotional factors playing a significant role. Empathy and machines’ understanding of human emotions are becoming integral to modern life [7]. Affective computing systems improve interaction experiences by understanding users’ psychological states, highlighting the need for a human-centered approach. MIT Media Lab research shows that affective computing enhances self-awareness, communication, and stress reduction [8]. As an emerging technology, affective computing offers real-time status detection and personalized services. It is aimed in this study to integrate affective computing into interpersonal interactions, providing timely assistance and decision-making to enhance the quality of people’s life.

Machine sensing methods fall into three categories: (1) Facial image-based sensing, such as eye trackers, which distinguish facial features from noise; (2) Voice and brainwave recognition, enhancing the reliability of physiological information beyond visual data; and (3) Object detection sensors, like infrared sensors, which use multi-sensing approaches to optimize human-machine interaction and feedback [9].

Affective computing analyzes human social behaviors and emotional information to provide appropriate feedback through human-machine interaction [10]. People use gestures and language for daily communication, and combining these modes offers high usability. Gestures can convey commands that are difficult to express verbally, simplifying and enhancing remote human-machine collaboration [11].

In summary, the aim of this study is to integrate affective computing with multi-sensing interactive interface technology to propose an emotion-aware interactive system characterized by pleasure, interactivity, and usability. The system will be tested in interpersonal interaction scenarios within smart living contexts to assess whether the criteria for usability, interactivity, and pleasantness are met.

1.2. Research Motivation

The COVID-19 pandemic has restricted global activities, boosting demand for online digitalization and shifting most interactions to virtual forms [12]. This has accelerated the development of physical digitization and contactless methods, overcoming time and space limitations. As the post-pandemic situation stabilizes, new interaction models, both proximate and remote, have emerged. Mixed-reality multi-sensing interfaces are subtly transforming daily life experiences.

Advances in artificial intelligence have strengthened the link between humans and smart technology, highlighting the growing importance of affective computing. There is increasing focusing on ensuring interactions to be friendly and enjoyable, with human-machine collaboration aimed at delivering warmer, higher-quality services.

According to Coherent Market Insights [13], the affective computing market, valued at $36.32 billion in 2021, is projected to reach $416.9 billion by 2030, with a compound annual growth rate of 31.5% from 2022 to 2030. The shift from rational to emotional user needs is driving leading technology companies to focus on the use of computational data for product design, a trend expected to continue.

Through interactions within specific contexts, novel situations and emotions can be evoked, creating fresh meanings. Human-machine interaction mechanisms are aimed to be established in this study using theories such as Interaction Ritual Chains and Situational Awareness. Leveraging the rise of affective computing technology, emotions are intended to be identified and recorded through interactions. Machine recognition of human emotions will be explored to enable diverse feedback tailored to different outcomes, enhancing personalized experiences and promoting greater self-awareness. It is noted that current designs of voice interaction systems utilizing multi-sensing technology are still limited.

Furthermore, it has been noted that emotional exchanges in Computer-Mediated Communication (CMC) environments are more frequent and clearer than in face-to-face (F2F) settings, with positive emotions being effectively amplified. Social media also enhances human interactions by offering low cost, flexible norms, and removing spatial and temporal constraints.

In summary, this study is aimed at developing a design system to enhance positive emotions in interpersonal interactions through the described mechanisms and to further investigate emotion transmission in communication. he integration of voice recognition into interaction patterns will also be explored to propose a novel system combining multi-sensing interactive interfaces with affective computing. The system will be applied to various smart living scenarios.

1.3. Research Problems and Purposes

In this study, the literature analysis has revealed that few interactive systems combine multi-sensing interaction with affective computing technologies. The issue of translating and reproducing users’ complex emotions by collecting emotional data from their interactions is to be explored. The goal is to help users better understand their own emotions through the system, making the experience more intuitive and relevant.

The questions studied in this research are as follows.

(1): How can the integration of multi-sensing interaction and affective computing technologies enhance interpersonal relationships?
(2): Does constructing an interactive system based on multi-sensing interaction and affective computing technologies strengthen positive interpersonal communication experiences and stimulate positive emotion transmission?
(3): Can the features of multi-sensing interaction and affective computing technologies improve the pleasantness of the interaction experience?

In this study, it is aimed to explore how to create pleasant interactive experiences for users through theoretical exploration and case analysis. An interactive system based on multi-sensing interfaces and affective computing technology is to be developed to enhance interpersonal interactions. Additionally, interaction mechanisms are to be investigated to promote positive emotional effects and examine emotion transmission in interpersonal interactions.

The research objectives are as follows.

(1): To analyze cases of multi-sensing interaction and affective computing, and summarize design principles for the proposed interactive system.
(2): To develop an emotion-aware interactive system that enhances positive interpersonal relationships.
(3): To evaluate whether the system effectively makes users feel pleasant and relaxed.

1.4. Research Scope and Limitations

In this study, the investigation will focus on interactive experiences using multi-sensing interfaces and affective computing. The scope and limitations are as follows:

(1): Voice recognition and Leap Motion gesture sensing are selected as the primary multi-sensing interfaces.
(2): Explore emotion recognition in interactive experiences will be explored, but the development of semantic analysis technologies will not be addressed.
(3): The designed multi-sensing system will be tested with users who have intact limbs, good auditory and visual abilities, and understand typical interpersonal interactions, specifically targeting active users aged 18 to 34.

1.5. Research Process

The research process, illustrated in Figure 1, is divided into five stages, with each stage’s details described below.

Research process of this study -

(1): Stage 1 - Research concept development:

(1.1) Understanding the research background and motivation.

(1.2) Defining the research purpose.

(2): Stage 2 - Literature review and case analysis:

Conducting analysis in three main areas: interpersonal interaction, affective computing, and multi-sensing interactive interfaces, as shown in Figure 2.

(3): Stage 3 – System planning and construction:

(3.1) Deriving design principles for multi-sensing interaction, including summarizing properties and proposing man-machine interaction architecture.

(3.2) Developing a prototype system using affinity diagrams.

(3.3) Constructing the final multi-sensing interactive system.

(4): Stage 4 – Exhibition and evaluation:

(4.1) Conducting public exhibition and experiments.

(4.2) Evaluating the proposed system by the methods of affinity diagrams and questionnaire survey.

(5): Stage 5 – Ending:

Drawing conclusions and making suggestions for future research.

Figure 2. Main areas for the literature review and research in this study.

2. Literature Review

As previously mentioned, the focus of this study is on interpersonal interaction, affective computing, and multi-sensing interactive interfaces, as shown in Figure 2. The literature review for each area begins with definitions and key issues (needs, mechanisms, developments) and is followed by a summary of findings that are used as a reference for the system design.

2.1. Interpersonal Interaction

A literature review is conducted focusing on emotional communication within interpersonal interactions, emphasizing interactive patterns and emotional perceptions between individuals. Research related to interpersonal interactions, both past and present, is examined to explore their content and characteristics. From this analysis, relevant design principles for the proposed system are derived.

2.1.1. Interpersonal Interaction Needs and Mechanisms

Interpersonal interaction, or interpersonal communication, involves the exchange of language, thoughts, and feelings between individuals. According to DeVito [14], interpersonal relationships serve four key functions: alleviating loneliness, securing stimulation, fulfilling needs, and providing contact for self-knowledge. These functions explain the motivation behind developing interpersonal relationships.

Interpersonal interaction is divided into verbal communication (e.g., face-to-face conversations, phone calls) and non-verbal communication (e.g., body language, facial expressions). Effective interaction is characterized by clear communication, active listening, and mutual understanding and respect. In daily life, communication generates symbols, leading to meaningful behaviors and reflecting new concepts and trends [15]. The processes of encoding (creating messages) and decoding (interpreting messages) are involved in communication [16]. According to Norman [17], the reliance on technology now spans information services such as communication, news, entertainment, education, and social interaction. Methods of communication, including symbols, sounds, text, and other perceivable signals, are used as valuable symbols in interpersonal interaction.

Today, multimedia communication has transformed interactions, connecting individuals across various locations and representing a new trend. Research shows that individuals aged 18 to 29 are the most active on social media, with approximately 84% of adults in this age group reporting use of at least one platform in 2021 [2,18]. Digital technologies have introduced various forms of interpersonal interaction, including real-time messaging apps, chatbots, and two-way audio-visual transmission.

Derks et al. [19] introduced computer-mediated communication (CMC), which uses computer networks to exchange digital messages and data. Their research indicates that emotional communication in CMC environments is more frequent and clearer than in face-to-face settings, potentially amplifying positive emotions [20]. Li et al. [21] found that social media, as a form of interpersonal interaction, addresses user needs such as finding common interests, anonymous participation, flexible norms, and overcoming spatial and temporal constraints, thus enhancing interactions.

In interpersonal communication, emotional contagion, or the “chameleon effect,” involves unconsciously imitating others’ emotions and behaviors. Research shows that individuals with higher empathy are more likely to exhibit this effect [22]. Emotional contagion can strengthen relationships and foster empathy, which aids in achieving shared goals [23].

Studies suggest that synchronized actions form a strong impression of interaction partners’ behaviors. Positive emotional contagion not only improves interactive task effectiveness but also enhances perspective-taking and empathy. Thus, a harmonious atmosphere, positive emotions, and attractiveness are crucial for future research [24]. High-quality connections (HQCs), as detailed in Table 1, are short-term, bidirectional interactions that foster engagement, openness, and positivity, with cognitive, emotional, and behavioral mechanisms being key [25].

In summary, understanding the connection between user motivations, emotions, and cognition is crucial for enhancing relationships and influencing continued system use. Cognitive interest criteria in interpersonal interactions will be discussed, and the derived design principles will be summarized in later sections.

2.1.2. Perceived Interest in Interpersonal Interaction

Jiarui et al. [26] highlighted that traditional research on interpersonal interactions often focused on single dimensions of human-to-human interaction. The integration of human, cognitive, and system interaction effects was attempted, and Davis’s technology acceptance model (TAM) was applied to explore perceived usefulness (PU) and perceived ease of use (PEOU) [27]. It was revealed that social network participation is driven by functional, social, psychological, and hedonic needs, demonstrating that emotions significantly impact interpersonal interaction intentions. Further investigation into emotional perception through the cognitive aspect of perceived interest will be conducted in this study.

In recent years, emotions like enjoyment, pleasure, and interest have been recognized as crucial for users’ intentions to continue using technology. Perceived enjoyment (PE) has been shown to significantly affect users’ attitudes and behavioral intentions towards technology, influencing actual usage behavior [28]. Venkatesh et al. [29] categorized enjoyment as intrinsic motivation and perceived usefulness (PU) as extrinsic, noting that enjoyment enhances ease-of-use and, in turn, perceived usefulness. Mun and Hwang [30] found that in enjoyable contexts, perceived enjoyment (PE) is a stronger predictor of usefulness (PU) than ease-of-use (PEOU).

Chung and Tan [31] identified key factors affecting users’ perceived enjoyment, with intrinsic motivation being central. They highlighted that content quality, ease of use, feedback diversity, usefulness, and pleasure are crucial for intrinsic motivation and should guide design processes. Brown et al. [32] introduced the situated cognition theory, which posits that cognition, thinking, and learning are context-dependent. This theory emphasizes the role of specific situations, spaces, and cultural contexts in shaping human knowledge and stresses the importance of designing interactions that align with socio-cultural contexts for effective cognitive and sensory management [33].

Furthermore, Schrepp et al. [34] compiled multimedia design guidelines, including: (1) minimizing visual clutter; (2) facilitating cognitive processes for information selection, organization, and integration; (3) enabling interaction with the environment and objects; (4) presenting multimodal information (text, images, sound); and (5) using animations appropriately. These guidelines offer preliminary directions for designing the interactive system in this research.

Collins [35] developed the interaction ritual chain theory by combining Goffman’s symbolic interaction theory with Durkheim’s social facts theory. This theory emphasizes “situational” perception and emotional aspects often neglected in social network studies. It suggests that rituals, through rhythmic interactions, focus users’ attention and emotional resonance, impacting the experience’s value. Collins [5] further highlighted that shared goals and emotions in interactions extend into daily life.

Bellocchi et al. [36] found that behavior, emotions, and cognition are interconnected. Interaction standards emerge from users’ collaboration and emotional resonance, with positive emotions indicating enjoyment and sustaining interactions. The interaction ritual chain theory’s uniqueness lies in its emphasis on users’ experiences and shared emotions, which foster group identification and enhance emotional energy, promoting positive emotions [37].

Patricia [38] challenged traditional views by noting that IoT technology has expanded digital social interactions, including communication and entertainment. Even without physical contact, emotional energy persists in virtual spaces like chatrooms, where users express emotions through images or text. This evolution of interaction ritual chain theory underscores the significance of feeling “understood” in strengthening relationships and improving interactions on social networks. Both situated cognition theory and interaction ritual chain theory enhance interpersonal interactions in virtual contexts.

Clark [39] noted that language uses reflect personal and social dimensions and emphasized that languages in interpersonal contexts involve collaborative narrative between speakers and listeners, with “understanding others” being central. Given the ambiguous nature of information in these contexts, understanding relies on various cues to extract key information.

Van Kleef [40] introduced the Emotions-as-Social-Information (EASI) model as illustrated in Figure 3, which posits that interpersonal emotional influences occur through reasoning processes or emotional stimuli. This model differentiates emotions based on the observer’s mood, indicating that discrete emotions (joy, anger, disgust, fear, sadness) serve a purposeful function, revealing the observer’s intentions or motivations in response to events.

Van Kleef et al. [41] validated the importance of emotions in situational contexts, emphasizing that decision-making in interpersonal interactions is often driven by emotions. Their research shows that emotions highlight event relevance and guide subsequent behaviors, including empathy and emotional contagion. Social interactions are identified as the primary contexts for expressing and sharing emotions, where individuals communicate their emotions and interpret those of others.

Wright and McCarthy [42] noted that most users rely on portable devices for information and entertainment, with instant messaging being a common mode of social interaction. Social apps like email, LinkedIn, Facebook, and Twitter use notification badges to prompt urgent responses, which can create anxiety and reduce interaction quality. The study suggests that slower messaging methods could alleviate stress and enhance communication by incorporating diverse perspectives, such as contextual, narrative, and autobiographical.

Hallnäs and Redström [43] argued that interaction design has evolved from creating efficient tools to developing experiences and smart technologies integrated into daily life. They introduced “slow technology,” which emphasizes moments for reflection and mental rest. Slow technology encourages thoughtful engagement and suggests that slower interactions, such as text-based communication, are crucial for reflective language processing and providing space for contemplation.

Therefore, in this study, an attempt is made to integrate the indicators of “Functionality,” “Usability,” and “Enjoyability” mentioned in the literature with concepts from theories such as situated cognition, interaction ritual chain, and emotions-as-social-information model. The system design principles derived are as follows:

(1): Effectiveness: Providing users with systems and functions that prioritize immediacy and convenience.
(2): Enjoyability: Considering interface diversity and emotional resonance to enhance user focusing and immersion.
(3): Simplicity: Offering a straightforward, intuitive, and natural user interface.
(4): Interactive Context: Recognizing the significance of multimedia virtual spaces in daily life and leveraging social interaction behaviors to stimulate users’ emotional awareness and behaviors.

2.1.3. A Summary

In this study, the broad definition of interpersonal interaction is explored. According to the literature surveyed, interactions primarily involve fixed, repetitive actions aimed at achieving expected goals. Scholars have identified that social network interactions fulfill functional, social, psychological, and hedonic needs. Design considerations must incorporate intrinsic user motivations. The situated cognition theory and interaction ritual chain theory underscore that in digital multimedia social spaces, users engage in real-time communication through voice and text, fostering emotional resonance. The Technology Acceptance Model suggests a positive relationship between usability and enjoyment, which will be detailed further in Section 4.

To design effective virtual social network interactions, incorporating these principles is essential. This approach aims to enhance users’ interaction experiences and foster continued engagement intentions. The study’s primary objective is to develop a system that enriches interpersonal interactions with novelty and enjoyment.

Key findings from this section include the following points.

(1): Emotional resonance and empathy: Interpersonal interactions evoke emotional resonance and empathy, fostering positive feelings through active communication.
(2): Intrinsic motivation in interaction contexts: Exploring intrinsic motivations within online contexts facilitates emotional exchanges and enjoyable flow experiences.
(3): Real-time interaction in digital multimedia spaces: These spaces enable interactions across formats, times, and locations, promoting continuous participation.
(4): Integration of cognitive and emotional aspects: Combining these aspects enhances system usability, addressing needs such as loneliness alleviation and self-understanding.
(5): Relationship between usability, enjoyment, and interaction quality: The system’s usability and enjoyment directly impact the quality of interpersonal interactions.

2.2. Affective Computing

2.2.1. Development of Affective Computing

The term affective computing was introduced by Picard [44] in 1995. This field combines psychology, cognitive science, and computer science to address emotional needs by detecting physiological data, also known as artificial emotional intelligence.

Norman [17] observed that while people use bodily movements to interact with machines, machines use sounds, tactile feedback, and displays to communicate. Advances in artificial intelligence have enabled machines to perform tasks once exclusive to humans. As machine learning and communication technologies advance, emotion-sensing technologies are increasingly used to interpret psychological states, leading to more human-like affective computing systems.

Picard [44] proposed a four-level framework for affective computing, which aims to develop systems that can recognize, interpret, process, and simulate human emotions:

(1): Recognizing Emotion: Analyzing physiological signals, voice, facial expressions, and text to determine users’ emotional states. This is the most widely applied level.
(2): Expressing Emotion: Enabling computers to communicate with users by expressing emotions. This level often involves simulation based on predefined settings, commonly seen in chatbots and virtual characters.
(3): Having Emotion: Allowing computers to possess emotions similar to humans, leading to emotional decision-making and actions. This level is still developing and requires careful management.
(4): Showing Emotional Intelligence: Computers can self-regulate, moderate, and manage their emotions, effectively functioning as “friendly artificial intelligence.”

Currently, most research in affective computing focuses on the first level, which is the foundational aspect of the field. The second level, involving the expression of emotions by computers, requires interfaces that simulate emotional expressions for effective user communication. Scholars recommend designing systems to recognize and express emotions akin to human interactions.

Early work by Breazeal and Scassellati [45] led to the development of the social robot Kismet, which mimicked human emotional communication through facial expressions and sounds. Kismet combined visual perception, attention, motivation, behavior, and interaction systems to create meaningful and intentional interactions, resembling human caregiving for infants.

Emotion is crucial in human cognitive processes and enhances both performance and decision-making capabilities in systems. Affective computing improves upon traditional interactions by making human-computer interfaces more accurate and engaging. According to Shukla et al. [46], emotion recognition is a trend that will become increasingly significant in the near future, offering personalized user experiences on social media. Despite this, the emotional aspects of social media behavior are often overlooked [47]. By enabling computers to recognize, categorize, and respond to human emotions, affective computing reduces frustration and aids in understanding both personal and others’ emotions [48].

Sloman [49] distinguished between primary emotions (e.g., fear, joy), which are instinctive and detectable through facial or bodily sensors, and secondary emotions, which are more complex and influenced by primary emotions. Secondary emotions, detected through semantic analysis, are often overlooked but are crucial in daily life. Understanding these emotions is essential, as detecting emotions solely through physiological appearance is insufficient.

Plutchik’s “Wheel of Emotions” model [50] pairs eight primary emotions in opposites: joy vs. sadness, anger vs. fear, trust vs. distrust, and surprise vs. anticipation. Secondary emotions arise from these primary ones. The model, illustrated in Figure 4a,b, represents emotions as a continuous process from stimulus events through cognition, sensation, impulsive action, and effects, providing a framework for understanding the relationships among various emotional states.

In subsequent research, Cambria et al. [51] expanded on Plutchik’s “Wheel of Emotions” model [50] by introducing the “Hourglass of Emotions” model (see Figure 5). This model arranges emotional states around four primary emotions, forming an hourglass shape that represents emotional intensity across four dimensions. The vertical axis shows the intensity of different emotional dimensions, while the horizontal axis reflects the intensity of emotions triggered by various stimuli. The Hourglass of Emotions model can assess human-computer interactions by measuring user pleasure (pleasantness), attention, sensitivity, and adaptability. It detects the current intensity of a user’s emotions through these dimensions.

Plutchik’s “Wheel of Emotions” model allows for observing primary emotions and their opposites while exploring various intensities and correlations between emotions. Feldman [52] notes that Russell’s early work [53] identified emotions based on “valence” and “arousal,” represented on horizontal and vertical axes, respectively, of an affect circumplex. The arousal dimension measures the level of emotional stimulation, ranging from calm and relaxed to excited and agitated. When combined with the valence dimension, which ranges from highly positive to highly negative, these dimensions effectively capture a user’s emotional state [53,56]. To understand emotion relationships, studies [50,54,55] have used similar scaling methods to position emotions on the circumplex, as shown in Figure 6. This approach involves selecting and explaining emotional states from a broad range of terms to aid in emotion validation and analysis, and is employed in this study.

Researchers often categorize emotional stimuli into four types based on valence and arousal dimensions as follows. (1) High Valence, High Arousal (HVHA): Positive and highly stimulating emotions. (2) High Valence, Low Arousal (HVLA): Positive but less stimulating emotions. (3) Low Valence, High Arousal (LVHA): Negative but highly stimulating emotions. (4) Low Valence, Low Arousal (LVLA): Negative and less stimulating emotions. Experiments expose participants to stimuli like sadness (negative, low arousal), calmness (positive, low arousal), anxiety (negative, high arousal), and happiness (positive, high arousal), plus a neutral condition, to observe and verify emotional state changes through physiological sensors [57,58,59,60].

Wang and Chen [61] created a system that visualizes and senses emotional states using physiological data. An EEG device monitors emotional states during drawing activities, translating EEG signals into various emotions (focus, relaxation, calmness, anxiety) and providing feedback where EEG signals include δ, θ, α, and β waves [62,63,64], as described in Table 2.

2.2.2. Related Cases of Affective Computing

The COVID-19 pandemic has accelerated the integration of affective computing into interpersonal interactions, making it a significant technological trend. This has resulted in more immediate, diverse, and innovative digital user experiences. This study analyzed several cases of affective computing in interpersonal interactions, comparing their content, forms, and applications, as detailed in Table 3. Additionally, the study examined the relationships between research motivations, content, and results in these cases, as summarized in Table 4.

In summary, emotions subtly shape daily life and relationships, with individuals constantly evaluating their emotional states through various systems. Affective computing leverages these emotions to assist with decision-making, emotional regulation, and stimulating emotional resonance. Boehner et al. [71] observed that emotions are subjective and arise from interactions with others. They categorized the effects of different media on emotional experiences into several areas: biological sensing, information generation, social and cultural factors, interaction behavior, and design and evaluation. They also offered related findings and recommendations as described in the following.

(1): Users construct emotional cognitive experiences based on information about interpersonal interactions.
(2): Emphasis may be placed on how users experience and understand emotions through system interactions.
(3): Emotional data should be situated within emotional dimensions to allow for free and rich emotional expression.
(4): The aim is to develop systems that provide users with meaningful emotional stimulation.
(5): Evaluation focuses on detecting and interpreting how users’ emotions change over time and with different stimuli.

According to Gartner [72], emotional artificial intelligence is a new addition to the emerging technology maturity curve, expected to peak within five to ten years. This technology is set to profoundly impact society. As an innovative human-computer interaction approach, affective computing provides design and evaluation strategies for co-creation and emotional experience, helping machines better understand human emotions and allowing individuals to better grasp their own and others’ emotions, thus enhancing decision-making and emotional expression.

2.2.3. A Summary

The key points from this literature review are as follows.

(1): Applications in communication: Affective computing is widely applied in interpersonal communication, with remote tools enhancing the detection of users’ emotional states.
(2): Empathy and emotional resonance: It enables computers to understand and express emotions, fostering empathy, relaxation, and focusing in users.
(3): Case study insights: Case studies commonly aim to improve negative emotions and enhance empathic communication, focusing on a better quality of life and human-centered experiences through techniques like semantic analysis.
(4): Encouraging interaction: System designs should promote user interaction and emotional awareness, offering cues that stimulate reflection without overly defining emotions.

It is observed that affective computing cannot rely on a single sensor to fully collect and interpret emotional data for users. Instead, integrating multiple sensor interfaces is necessary for comprehensive emotional data collection and detection. The next step will be to explore interactive interfaces with multiple types of sensors.

2.3. Multi-Sensing Interactive Interfaces

2.3.1. Needs of Multi-Sensing Interactive Interfaces

With advanced sensors now common in electronic devices, a wealth of emotional information is available in social network communications, including audio and video. Poria et al. [73] found that interactions have shifted from single to multi-modal approaches with the rise of the Internet. While most research focuses on audiovisual sensing, there is limited work combining semantic analysis with audiovisual data. Poria et al. [74] highlight that natural human communication is inherently multi-modal, involving text, sound, and visuals. Effective research should integrate these modalities to capture and convey semantic and emotional information seamlessly.

Fang et al. [75] identified multimodal sensing and integration with human sensory inputs as a promising research direction. Single-modal sensors can be prone to errors and noise, making multi-sensory feedback a valuable complement. They noted that finger and wrist angle detection in hand movement technology is increasingly popular. In smart living applications with many interconnected devices, this integration enhances user convenience and flexibility. The study emphasizes the importance of clear gesture recognition, recommending that similar gestures be differentiated to prevent errors and that meaningless gestures be redefined for improved user experience [76].

Due and Licoppe [77] highlight that gestures act as indicative and symbolic signs, with their meanings shaped by context. Their research suggests that multimodal sensing, informed by Gestalt psychology, enhances interaction by integrating users’ past experiences with the current gesture shape and position. This approach fosters more intuitive interactions, even without verbal communication.

Ferscha et al. [78] support that gestures offer intuitive command interactions. They emphasize the need for standardized and flexible gesture commands, focusing on directional position sensing to enhance naturalness and immersion. Designing paired commands (e.g., start/pause, left/right) and handling complex hand gestures improves interaction efficiency and integration across different sensors.

To express thoughts and emotions, people rely on sharing, communication, and message transmission, necessitating accurate sensor data capture. Jonell et al. [79] introduced a multi-sensing interaction framework that integrates voice, motion, and other sensory inputs (see Figure 7). This modular design approach involves breaking the system into units that process sensor data sequentially. The framework includes four layers: sensor API module, sensor data preprocessing module, information processing module, and end user. Voice data, captured by a microphone sensor, is processed by the preprocessor module for behavior detection or visual processing, and then delivered to the end user. The results show that a modular system enhances flexibility and intuitiveness in multi-sensing interactions.

Ueng and Chen [80] proposed a multimodal system using computer vision for environmental interaction. They observed that users typically interact with screens from 80–90 centimeters away and perform hand gestures within 50 centimeters of the camera. Leap Motion, known for its high gesture detection capabilities, is frequently used in these contexts and often combined with voice sensors for a seamless, user-friendly interface in smart home applications.

Yasen [81] introduced an application that translates gestures into voice commands, emphasizing both touch and non-touch interactions. Non-touch gesture recognition, which converts finger movements into commands, allows remote control of devices and avoids touch-based infections. This can be achieved through handheld or wearable sensors or vision-based gesture recognition, facilitating hands-free interaction.

In 1980, Bolt [82] explored natural user interfaces integrating speech and gesture inputs. The study found that combining gestures with speech improved system interpretation accuracy and user experience. It suggested minimizing the need for pauses during speech commands by providing sensory smart prompts to help users understand system processing, thus reducing anxiety.

Billinghurst [83] argued that speech and gestures complement each other, with speech suited for descriptive tasks and gestures for direct manipulation. Most users preferred combining both methods, with 71% using them simultaneously, compared to 13% using only speech and 16% using only gestures. This multimodal approach increased recognition accuracy and reduced command completion time, enhancing user focus on the screen by up to 108%.

The design of user interfaces remains an interdisciplinary challenge. Jalil [84] proposed a framework for “Intelligent User Interfaces” (IUI), shown in Figure 8, aimed at improving performance, emotional engagement, and naturalness in human-computer interactions. This framework employs computer vision and speech recognition technologies to understand natural language. The research highlights that sensing human emotions will be pivotal in future intelligent user interfaces, offering immersive experiences, responsive interactions, and aiding in decision-making and contextual guidance.

Song et al. [85] emphasize that sensor-based interactive interfaces must become more human-centered, with performance characteristics like sensitivity, flexibility, and measurement range being crucial for future applications.

Key points from this literature review include the following.

(1): Modular design framework: Modular design enhances flexibility and provides a more intuitive interaction experience.
(2): Design principles: Effective gesture commands often rely on directional sensors to increase interaction naturalness and immersion. Emphasis is placed on paired command combinations and minimizing the use of pause commands during interactions.
(3): Integration of voice and gesture: The combination of non-contact multi-sensing interfaces (voice and gesture) within the Intelligent User Interfaces (IUI) framework supports the development of an innovative and engaging multi-sensing interactive affective computing system.

Summary details of the multi-sensing interface design principles are provided in Table 5.

Based on the above discussion, it is tried in this study to integrate multi-sensing interactive interfaces with affective computing to explore users’ cognitive emotions. The aim is to create an efficient and natural interaction experience. By analyzing multi-sensing designs and referencing interface design principles, in the study efforts will be made to seek a better design of an interactive system that aligns with these principles. The ultimate goal is to design a suitable affective computing sensor system.

2.3.2. Related Cases of Multi-Sensing Interactive Interfaces

Multi-sensing interfaces overcome the limitations of single-input systems by integrating inputs from multiple devices, enhancing interaction speed, accuracy, and naturalness. This shift from traditional mice, keyboards, and touchpads to simultaneous inputs like speech, text, and gestures increases focusing and immersion. Assistive applications are increasingly important, offering contextual recommendations and improving quality of life in a user-friendly way [86].

In this study, focuses are put on combining natural gesture control with voice recognition. A comparative table (Table 6) has been created to evaluate the characteristics and application areas of six existing cases of multi-sensing interactive interfaces, providing a reference for designing the affective computing sensor system.

2.3.3. A Summary

In smart living applications, integrating multi-sensing inputs is a novel area of human-computer interaction research. In this study, the focus is put on combining “non-contact gesture interfaces” and “voice interfaces” to enhance smart living applications. The key points of the previous literature review are summarized as follows.

(1): Enhanced accuracy and efficiency: Multi-sensing inputs improve system accuracy and efficiency, leading to greater focusing and immersion.
(2): User preferences: Users generally favor interfaces that combine non-contact gestures and voice control, with voice interfaces boosting user concentration.
(3): Intelligent user Interfaces: The IUI framework can enhance performance, emotional engagement, and naturalness in interactions by sensing emotional information.
(4): Growing importance of assistive features: Assistive features are becoming more critical in multi-sensing interfaces. In this study, it will be tried to explore integrating affective computing into interaction interfaces to improve user quality of life and evaluate user experience in a simple, intuitive manner.

2.4. Concluding Remarks of the Literature Review

In this section, the exploration of emotional mechanisms and behaviors in interpersonal interactions is presented. Key mechanisms from research are summarized, and a review of literature on affective computing and multi-sensing interactive interfaces is conducted, with existing cases analyzed to establish design principles for this study.

The primary aim in this study is to integrate multi-sensing interactive interfaces with affective computing to enhance interpersonal communication. The design principles for this integration are as follows:

(1): Intentions of interpersonal interaction: Goals include alleviating loneliness, seeking stimulation, satisfying needs, and enhancing self-understanding. In this study, it is aimed to evoke emotional resonance and generate positive experiences by integrating cognitive processes and context, measuring pleasantness, attention, and emotional intensity based on the emotional hourglass model.
(2): Benefits of affective computing: Affective computing aids in real-time emotional regulation and interpersonal issue resolution, while also enhancing decision-making capabilities.
(3): Advantages of multi-sensing interactive interfaces: Combining voice and gestures in multi-sensing interfaces surpasses traditional methods, improving focusing and resolving ambiguous inputs through contextual semantic analysis. The interface’s effectiveness is evaluated based on sensitivity, flexibility, and measurement range, enhancing system accuracy and effectiveness.
(4): Integration and evaluation of sensor data: Integrating multiple sensors to capture user input is assessed using the indicators of effectiveness, simplicity, and pleasantness to highlight the importance of user involvement in the interaction context.

3. Methodology

3.1. Selection of Research Methods

The research methods used in this study are the affinity diagram method, prototype development, indirect observation, and questionnaire surveys. The applications of these methods in this study are illustrated in Figure 9.

The research process began with a literature review to analyze interpersonal interactions, affective computing, and multimodal interactive interfaces, which led to the formulation of the initial design principles for this study. The affinity diagram method was then applied to discuss and refine these principles based on user feedback. A prototype system was developed accordingly and tested through public demonstrations. Finally, the indirect observation and questionnaire survey methods were used to gather insights on user behavior and emotions, which informed further analysis and prototype refinement.

3.2. The Affinity Diagram Method

3.2.1. Idea of Using the Affinity Diagram Method

The affinity diagram method, or KJ Method, introduced by Japanese ethnologist Jiro Kawakita in 1960 [93], is widely used in human-computer interaction research. This technique organizes diverse and unstructured qualitative data related to products, processes, or complex issues by grouping observations based on similarities or dependencies [93,94,95]. Originally applied to synthesize field survey data and generate new hypotheses, the method is now commonly used to analyze contextual inquiry data and guide the design process, providing a qualitative foundation for decision-making [96,97].

The affinity diagram method, utilized primarily in the early stages of the design process, is employed in this study to define and analyze interactive prototypes. Initially, the target user group is identified, and user personas are developed to guide core research needs. Participants then generate descriptive attributes for the target user group on sticky notes, sharing their insights on these users [98,99].

Based on cutting-edge desktop research and literature, six target users utilized the AEIOU framework—focusing on the topics of Activity, Environment, Interaction, Object, and User [100]—to conduct “design thinking” and explore innovative solutions. The clustered data from this process is organized and presented in this study. Qualitative insights, including pain points and core needs of the target user profiles, were identified. Additionally, persona and empathy map design methods were employed to further structure the context and develop detailed user profiles and scenarios, as described in the following sections.

(1) Persona -

The persona concept, introduced by Cooper [101], involves creating virtual characters to represent user groups with specific attributes. Since then, many leading companies have adopted this method. Personas offer a user-centric view, bridging the gap between designers and users. Cooper and Reimann [102] highlighted three key benefits: (1) providing human-centered insights that reduce the designer-user gap, (2) fostering shared understanding and consensus in teams, and (3) enabling precise targeting of user groups, thus improving development efficiency.

(2) Empathy map -

The empathy map, a user-centered design (UCD) method, helps clearly understand user interactions and core needs by empathizing with their situations. It supports software development by validating features through user perspectives [103]. In the design thinking process, the empathy stage involves analyzing stakeholder perspectives to grasp pain points, needs, and expectations. This creates visual clues and application trends based on contextual data [104]. Ferreira et al. [105] noted that empathy maps are well-regarded in software engineering. The map includes six sections: (1) See: what the user observes; (2) Say & Do: user’s verbal expressions and behaviors; (3) Think & Feel: internal experiences and emotions; (4) Hear: environmental influences; (5) Pain: frustrations or difficulties; and (6) Gain: user goals. Empathy maps are effective for depicting and creating target user personas.

3.2.2. Application in This Study and Brainstorming Result

This study utilizes the Affinity Diagram Method to consolidate information from personas and empathy maps, integrating them with the “Job-to-be-Done” (JTBD) framework. The JTBD framework, originating from innovation and entrepreneurship, explores the cognitive psychology and emotional states that drive user behavior. Its goal is to uncover users’ primary motivations and core needs, which is essential for creating designs that are both appropriate and satisfying. By analyzing who (users), what (tasks), how (methods), why (motivation), when (timing), and where (context), the JTBD framework facilitates a thorough contextual investigation. JTBD focuses on three main indicators: (1) Functional: How users perform or approach tasks; (2) Social: The impact of user interactions with others; and (3) Emotional: Users’ feelings and experiences during tasks.

These indicators help in developing a North Star Metric to understand pain points deeply and identify innovative solutions. Overall, comprehending users’ basic needs and core tasks enables the creation of a user-centered design (UCD) experience that meets expectations [106,107,108,109,110,111,112]. In this study, the JTBD framework is used to categorize brainstorming data, while the affinity diagram method consolidates extensive research data for user journey mapping and scenario storytelling.

In this study, six target users formed a co-creation group to brainstorm and use the affinity diagram method for organizing data. This process aimed to gather diverse opinions, ideas, and experiences in a non-judgmental manner, fostering creative thinking and guiding collaborative actions toward consensus. Three core themes emerged from this investigation as described in the following.

(1) Daily life relevance: Ensuring shared understanding among users.

(2) Open and flexible discussion space: Stimulating user inspiration.

(3) Emotional resonance: Providing a comforting and relaxing experience.

The brainstorming results helped shape user profiles and scenarios, focusing on students aged 18 and above. By leveraging these core themes, the study aims to enhance interpersonal communication. The research introduces the theme “what superpowers users wish to have,” allowing users to freely express their opinions or address issues they want to solve with superpowers. The interactive system developed provides a platform for users to express emotions and explore emotional exchanges in interpersonal interactions.

3.3. The Prototype Development Method

Bernard (1984) defined prototype design as a modeling method that efficiently captures user feedback and defines strategies for user needs with minimal time and cost. This method supports iterative refinements, enhancing system development maturity. Hornbæk and Stage [113] highlighted user testing as a common method for evaluating early prototypes, where users interact with the system to provide feedback, guiding design improvements. According to Bernhaupt et al. [114], evaluation testing aims to gather user feedback on usability and experiences, facilitating iterative design refinements. Bernard [115] further noted that this approach improves system output accuracy and aids decision-making, thus supporting systematic prototype development [116].

In this study, five steps based on multi-sensing interactive interfaces and affective computing are taken to develop a prototype system, as described in the following and shown in Figure 10.

(1)

Defining stage - Defining the design needs for the system to be constructed.

(2)

Development stage –

(2.1): Designing a prototype system based on the techniques of multi-sensing interactive interfaces and affective computing.
(2.2): Conducting user testing of the prototype system.

(3)

Evaluation stage –

(3.1): Conducting experiment analysis.
(3.2): Iterating to Step (2.1) to modify the system whenever necessary.

(4)

Ending stage - Completing the system development

3.4. Indirect Observation Method

Turner [117] highlights that positivist theory, founded by Auguste Comte, emphasizes objective observation as a core research method. Williamson [118] describes observation as both a research method and a data collection technique, while Baker [119] notes its complexity due to the varied roles and techniques required from researchers. Regardless of their involvement level, researchers must uphold objectivity to ensure accurate data collection and analysis. Validity is categorized into descriptive, interpretive, and theoretical types, with theoretical validity assessing how well theoretical explanations align with the data.

Bernhaupt et al. [114] found that objective observation can lead to deeper analysis and iterative system refinements. Future research may leverage observational data (such as images and sounds) or physiological sensing (like eye-tracking, skin conductivity, and EEG) to validate study effectiveness. Advancements in technology enhance indirect observation by improving data recording, communication, and storage, thus providing a flexible and rigorous method for capturing user insights and validating hypotheses [120].

Indirect observation offers the advantage of capturing users’ natural states without the need for researchers to be present for extended periods or intervene directly. This approach yields authentic, reliable data and is suitable for collecting a wide range of information, providing a more objective, comprehensive, and diverse perspective.

Indirect observation is employed in this study as a method for data collection, using natural and non-intrusive sensing of users’ physiological emotional characteristics during system interactions. It involves collecting and analyzing physiological emotional data such as semantic and EEG information. This method not only aids in understanding emotional transmission trends in interpersonal interaction contexts but also strengthens and validates the research.

3.5. Questionnaire Survey Method

3.5.1. Ideas and Applications of the Method

The questionnaire survey method is widely used for its reliability and effectiveness in collecting key data and information [121]. Designing surveys requires clear, straightforward language and avoids ambiguous or leading questions to ensure validity [122]. This method helps explore users’ views and attitudes, guiding subsequent prototype revisions. In this study, a questionnaire will be administered during the public exhibition period, utilizing a five-point Likert scale for evaluation.

The System Usability Scale (SUS), developed by Brooke [123], is a flexible Likert-type scale with ten straightforward items. It assesses user satisfaction with systems based on agreement levels and can be used for minor adjustments or comparative analysis of different systems. The SUS provides a quick measure of usability, focusing on effectiveness and efficiency [124,125].

In human-computer interaction research, Zhang and Adipat [126] identified “effectiveness,” “simplicity,” and “enjoyment” as key usability metrics for evaluating user experience with systems. Davis [27] highlighted a correlation between perceived usefulness and perceived enjoyment in the technology acceptance model. Additionally, Schrepp et al. [34] linked experiences of pleasure, excitement, and novelty to perceived enjoyment, suggesting that emotions can reflect aspects such as system effectiveness, reliability, and interactivity.

Tiger [127] identified four categories of pleasure derived from products: “physiological pleasure,” “social pleasure,” “psychological pleasure,” and “intellectual pleasure,” providing a nuanced framework for understanding different aspects of pleasurable experiences. Jordan [128] emphasized three challenges in studying pleasure: understanding users and their needs, linking product characteristics to pleasurable qualities, and developing quantifiable metrics for measuring pleasure. As shown in Figure 11, Jordan [129] also proposed a hierarchical structure of user needs—functionality, usability, and pleasure—where functionality forms the base, usability is the next level, and pleasure represents the highest level, emerging once usability needs are met.

3.5.2. Purpose of Survey and Question Design

It is aimed in this study to assess through surveys how users enhance interpersonal interactions, contextual awareness, and emotional resonance with the system. Hassenzahl [130] proposed the Hedonic Quality Model, emphasizing that satisfaction arises from subjective experiences of design, with the relationship between usability and hedonic quality influenced by design and context. Hernández-Jorge et al. [131] analyzed emotional communication and found that creating a positive emotional atmosphere, caring about others’ feelings, and active listening are crucial for emotional interaction and communication. Expert validity may be employed to assess the survey’s effectiveness, focusing on content validity. This process involves selecting at least two knowledgeable experts, distributing the survey to them, collecting their feedback, and revising the survey based on their recommendations [132,133].

In summary, the dimensions of “system usability,” “interactive experience evaluation,” and “pleasure experience” are utilized for designing the questions used in the questionnaire survey in this study. The dimensions and questions of each dimension are detailed in Table 7.

4. System Design

In this study, an interactive system called “Emotion Drift” is designed, utilizing multi-sensing interfaces and affective computing. This system explores emotional communication in interpersonal interactions by integrating inputs from various non-contact methods, including gestures and voice, extending beyond traditional input techniques. By combining affective computing technology, the system provides diverse audiovisual feedback based on the user’s emotional state. It offers a user experience that balances pleasure, interactivity, and usability through natural and intuitive interactions. Further details are provided in the following sections.

4.1. The Design Concept of the Proposed System

The COVID-19 pandemic has accelerated the global shift towards virtual social media interactions to minimize physical contact between individuals. As a result, digital interpersonal communication has become a primary mode of exchange in modern times. However, emotions are often conveyed alongside shared text and audio, leading to emotional resonance. To address and mitigate negative emotions that may arise during interactions, the proposed system has been designed with an emotional feedback mechanism, providing visualized outputs that empathize with users’ feelings.

It is aimed in this study to offer timely emotional audiovisual feedback to users and observe emotional rendering phenomena during interactions. The goal is to improve social relationships by alleviating negative emotions through interpersonal sharing and communication, thereby enhancing emotional regulation, empathy, and strengthening relationships. The system “Emotion Drift” is designed to help users achieve these objectives, offering a high-quality and humane experience. An illustration of the proposed system and the performance environment is shown in Figure 12.

The proposed Emotion Drift System constructs a world for asynchronous interpersonal communication, where emotions are thought to flow like “drifting bottles” — rising and falling with the intensity of winds and storms. Users can navigate imaginatively to isolated islands by “rowing” and pick up drifting bottles left by previous users to engage in deeper exchanges and interactions of emotions. These bottles symbolize a safe and private space for users to express their innermost thoughts.

However, because emotional communication should be fluid and historically recorded, the proposed Emotion Drift System is positioned as an emotional perception system. It invites users to enter others’ emotional worlds, sharing voices and expressing feelings together.

4.2. Design of the Multi-Sensing Interaction Process for the Proposed System

4.2.1. Idea of the Design

In this section, the interaction process using the proposed Emotion Drift System is explained. Users can awaken the system screen from standby mode through gestures, creating a pleasant and relaxing environment. While using the system, the user is imagined to be navigating a boat on a calm sea or river. The user controls the direction of the boat by rowing to explore the water space and pick up a drift bottle at indicated locations based on guidance.

Inside the bottle, the user finds four letters left by the previous user, each containing an answer to a question about “becoming a person with superpower.” The current user reads each letter, which is presented in both text and audio formats, and experiences the previous user’s thoughts on the question, as if interacting directly with the previous user. This process is repeated for all four letters.

Additionally, after listening to the previous user’s audio content, the current user can share his/her own thoughts with the drifting bottle, representing the previous user, to express understanding and empathy for the previous user’s feelings. The system also visualizes the current user’s emotional information, measured with an EEG device, creating graphic illustrations and textual meanings for review, thereby offering a user-centered interactive experience.

4.2.2. Detailed Description of the Interaction Process

More specifically, the interactive flow of the system is mainly divided into five stages: the guidance stage, the experience stage, the sharing input stage, the emotional review and output stage, and finally the ending stage, as illustrated by Table 8 and described in detail in the following.

(1)

Guidance Stage:

(1.1): System Awakening - The user activates the proposed interactive system by approaching it. Audiovisual effects are used to bring the “emotional” drifting sailing journey to life on the TV screen through gesture control and interaction with objects.
(1.2): Instruction learning - The user reviews the interactive scene and process instructions.

(2)

Experience Stage:

(2.1): Sailing forward - The user rows the boat forward using guided gestures to explore the interactive experience.
(2.2): Finding a drafting bottle - The user locates a bottle drifting on the water, containing a letter presumably left by the previous user.
(2.3): Picking up the bottle – Using appropriate gestures, the user retrieves the bottle, which contains four letters. Each letter includes answers to questions about “becoming a person with superpowers,” provided by the previous user.
(2.4): Taking out a letter – The user imaginatively extracts one of the letters from the drift bottle and displays it on the interface.
(2.5): Reading the letter content - Through gesture-based operations on the interface, the user reads the letter’s content, listening to the spoken message to experience and empathize with the emotions expressed by the previous user.
(2.6): Repeating message reading four times – The user repeats Steps (2.4) and (2.5) until all four letters have been read.

(3)

Sharing Input Stage:

(3.1): Activating voice recording - The user presses a voice-control button on the interface using gestures to initiate the system’s recording mode.
(3.2): Answering a question and recording – The user verbally responds to each of the four questions, sharing their emotions interactively with the previous user. The system records these responses for playback.
(3.3): Transforming the message into text - The recorded voice is automatically converted into text in real time by speech-to-text technology, which is then displayed on the interface.

(4)

Emotional Review and Output Stage:

(4.1): Semantic analysis and emotion presentation - Using natural language processing, the system analyzes the text generated in the previous stage along with the brainwave data collected from the EEG. This analysis identifies the user’s emotions and related parameters, which are then displayed on the interface for review, including a text description and four parameter values.
(4.2): Repeating answer recording and analysis four times – The user repeats the tasks outlined in Steps (3.1) through (3.3) and (4.1) until all four questions are answered.

(5)

Ending Stage:

(5.1): Showing the conclusion of emotion analysis – The system presents a summary of the user’s four emotional states through textual descriptions for review.
(5.2): Drafting a bottle with message letters – The user writes responses to the questions on letters, places them in a bottle, and releases it into the water for the next user.

Table 8. The process of user interaction on the proposed “Emotion Drift System.

Stage	Step theme	User’s interactive action	System’s operation
(1) Guidance Stage	(1.1) System wakening	The user awakens the system before the Leap Motion by hand waving gesture.	The Leap Motion in the system senses and recognizes the user’s gesture and moves the boat forward
(1) Guidance Stage	(1.2) Instruction learning	The user views the interactive scene and the process instruction.	The system displays the process instructions on the TV screen.
(2) Experience Stage	(2.1) Sailing forward	The user sails the boat forward by rowing gesture.	The system shows the rowing action of the user’s two hands and the forward moving of the boat.
	(2.2) Finding a drafting bottle	The user finds out a drafting bottle on the water.	The system displays the bottle in front of the boat.
	(2.3) Picking up the bottle	The user picks up by two-hand gestures the drafting bottle on the water with a letter left inside by the previous user.	The system shows two hands holding the bottle.
	(2.4) Taking out a letter	The user takes out a letter inside the bottle by pushing a button on the interface.	The system displays the cirresponding question and four “push buttons” on the interface shown on the TV screen.
	(2.5) Reading the letter content	The user reads the dsiplayed message on the letter by listening to the message spoken audibly.	The system plays the audio of the message recorded previously whne the previous user used the system.
	(2.6) Repeating message reading four times	The user does the same task of Steps (2.4) and (2.5) until all four letters are read.	The system repeats the operations done in Steps (2.4) and (2.5) four times as requested by the user.
(3) Sharing Input Stage	(3.1) Activating voice recording	The user pushes the “voice-recording button” on the interface by gestures to activate the system’s recording status	The system gets into the voice-recording state.
	(3.2) Answering a question and recording	The user responds by speaking out his/her thoughts to answer a question after pushing a corresponding buttom on the interfce.	The system records the voice of the user’s answer that can be re-played for review if the user push the “listening button.”
	(3.3) Transforming the message into text	The user views the text yielded by the system, which is what he/she said as the answer to the question.	The system transforms the recorded user’s verbal statements into text and displays the result on the interface.
(4) Emotion Review and Output Stage	(4.1) Semantic analysis and emotion presentation	The user views the emotion analysis result on the TV screen, including a text descriprtion and four parameter values.	By natural language processing, the system analyzes the text and brainwave data to identify the user’s emotions and present them on the screen.
(4) Emotion Review and Output Stage	(4.2) Repeating answer recording and analysis four times	The user conducts the tasks of Steps (3.1) through (3.3) and (4.1) repeatedly until all four questions are answered.	The system repeats the operations done in Steps (3.1) through (3.3) and (4.1) four times as requested by the user.
(5) Ending Stage	(5.1) Showing the conclusion of emotion analysis	The user views the concluding textual listing of the system’s analysis of his/her emotion.	The system presents a concluding listing of the textual descriptions of the user’s four types of emotions.
(5) Ending Stage	(5.2) Drafting a bottle with message letters	The user writes imaginatively the answers on letters, puts them into a bottle, and drafts it away for the next user.	The system shows the image of drafting the bottle on the TV display.

4.2.3. Questions to Answer as Messages for User Interactions

The previously-mentioned four questions to be answered by system users as messages for interactions (see Stage (3) above) are listed in Table 9. The four questions are related as a sequence with continuity in meanings:

Introduction → Development → Turning → Conclusion.

Table 9. Questions to answer as messages for user interactions.

No.	Property	Question content
1	Introduction	What superpower did the previous person share? What are your thoughts on it?
2	Development	What do you think are the good and bad aspects of it?
3	Turning	What superpower would you want and why?
4	Conclusion	How would you recommend this superpower to friends and family?

4.3. System Architecture and Employed Technologies

4.3.1. Overview of the Architecture

The affective computing functions of the Emotion Drift System rely on gesture and voice recognition for multi-sensing input, and use speech-to-text technology and brainwave analysis for emotion detection and visualization. The architecture for these functions is detailed in Figure 13.

The computational process of the proposed system is divided into three main components:

Gesture Recognition: The Leap Motion sensor detects hand gestures to control objects within the virtual interface of the Leap Motion software.
Voice Recognition: The Google Speech API processes voice input, converting speech to text in real time (Speech-to-Text, STT). It also employs natural language processing (NLP) to extract semantic meanings and performs sentiment analysis to assess the emotional content of the user’s speech.
Brainwave Monitoring: The Mind Sensor EEG headset captures brainwave activity, providing data for observing and visualizing the user’s emotional state.

These components are integrated using the Unity game engine, which provides audiovisual feedback based on detected emotions, as shown in Figure 13. More details will be described in the following sections.

4.3.2. Leap Motion Hand Gesture Sensor

The Leap Motion hand gesture sensor is a device designed specifically for gesture control and hand motion tracking. It uses a combination of infrared LEDs and cameras to detect and track hand movements with high accuracy. The sensor captures hand gestures and movements within a range of approximately 20 to 60 centimeters above the device. The real-time gesture data captured are then converted into information that the computer can interpret, providing users with a natural and intuitive interactive experience.

More specifically, the Leap Motion sensor detects gestures in spatial positions and movement states along three main axes: X, Y, and Z. The horizontal X-axis corresponds to lateral movements; moving the hand left or right changes the X-axis value accordingly. For example, moving the hand to the right increases the X-axis value, while moving it to the left decreases it. The vertical Y-axis is associated with up and down movements. The Z-axis represents depth, relating to forward and backward movements. When the hand moves closer to or farther from the sensor, the Z-axis value changes: moving the hand closer decreases the Z-axis value, and moving it farther away increases it, as illustrated in Figure 14.

In summary, the Leap Motion sensor uses infrared cameras to detect hand gestures in real-time in the physical world (as shown by the green coordinate lines in Figure 15). These gestures are then mapped to virtual world coordinates in the Unity game engine to control interactive objects (as shown by the red coordinate lines in Figure 15).

The system design of this study categorizes gesture commands into five types: “Move Forward (both hands move back and forth),” “Move Left (right hand moves back and forth),” “Move Right (left hand moves back and forth),” “Select Click (extend the index finger to touch the object),” and “Grab Object (open the palm to close the fist towards the object).” These gestures are detailed in Table 10. The development design and judgment logic of the Leap Motion sensor will be further elaborated in later sections.

Obviously, the determination of gesture commands depends on the recognition of the palm shape of the hand, whose skeleton consists of five fingers as shown in Figure 16. By calculating the distance between the fingertip position and the palm position, the current bend of a single finger can be determined. A threshold is set to indicate whether the finger is currently in a straight or bent state. The program developed in this study makes decisions in the aspect in the following way.

(1): If four fingers on one hand are bent, it indicates a fist gesture.
(2): If four fingers are straight, it indicates an open-hand gesture.
(3): By using the codes of the OnTrigger collision event offered by Unity, when the arm is detected to contact with an interactive object, collision detection is activated.
(4): When the object is grabbable, making a fist will allow the object to be picked up, and an open hand will allow the object to be placed down.
(5): When the object is an interactive button, the button will be pressed and moved by the arm. This can be cross-referenced with the gesture command table (Table 10) mentioned previously.

Figure 16. Leap Motion hand skeleton description (image source: https://www.ultraleap.com).

4.3.3. Google Speech API

Google Cloud’s NLP technology for sentiment analysis is utilized in this study, employing its open-source machine learning applications. The NLP analysis is performed via a REST API, which supports scalable, high-performance, and reliable communication while being easy to implement and modify. High levels of visualization and cross-platform transferability are provided. Data and service requests are sent to other applications by calling the API. The Google Cloud API service requires JSON data format for cloud requests, with sentiment parameters being extracted through the uploading of voice-related file formats.

The user’s speech is captured using a microphone and encoded for storage. The speech information is then converted into text data via the Google Cloud API service. Sentiment information from the text data is subsequently analyzed using the Natural Language Processing (NLP) technology provided by the same Google Cloud API service, as shown in Figure 17. During this process, NLP techniques are used to analyze the text data to extract two sentiment parameters: score and magnitude. The score reflects the overall emotion expressed in the text, while the magnitude measures the intensity of the emotional content, which is often proportional to the length of the text.

The Natural Language Processing (NLP) technology for sentiment analysis, provided by Google Cloud’s open-source machine learning applications, is utilized in this study. The analysis is conducted using a Representational State Transfer (REST) API architecture, which supports large-scale, high-performance, and reliable communication while being easy to implement and modify. High levels of visualization and cross-platform transferability are supported. The API allows data and service requests to be sent to other applications. The Google Cloud API service requires JSON format for cloud requests, enabling sentiment parameters to be extracted from uploaded voice-related file formats.

The score value (hereafter referred to as the S value) ranges from −1.0 to +1.0, representing the overall emotional tone of the text data from the speech. An S value greater than 0 indicates a predominantly positive emotion, while a score less than 0 indicates a predominantly negative emotion. The magnitude value (hereinafter referred to as the M value) ranges from 0.0 to +∞ (positive infinity) and reflects the quantity of emotional content in the text. The presence of both positive and negative emotional words in the text increases the M value. Table 11 presents examples of text-described emotional states along with their corresponding numerical S and M values, as obtained from the sentiment and semantics analyses shown in Figure 17.

4.3.4. Mind Sensor EEG

In this study, the Mind Sensor lightweight EEG device, as shown in Figure 18, is used to measure and record brain activity. This device is commonly employed in healthcare, wellness, gaming, education, and sports. It operates by using electrodes to detect electrical signals from the brain, reflecting various mental states such as concentration, relaxation, and excitement.

Before starting the interaction with the proposed Emotion Drift System, each participant was required to wear the EEG equipment described earlier. The equipment uses Bluetooth Low Energy (BLE) technology for wireless communication. Once the EEG device connects successfully to Unity, it provides several metrics: the “signal quality value” (denoted as sq) to indicate current signal stability, the “attention value” (denoted as a) to measure the participant’s level of focus, and the “meditation value” (denoted as m) to gauge the participant’s level of relaxation. The system not only records the user’s brainwave states but also quantifies the attention and meditation values, a and m, on a scale from 1 to 100 for emotional visualization purposes.

After experiencing the interaction on the proposed system, the current user will obtain the semantic values M and S as well as the EEG data values m and a. In this study, the EEG device outputs meditation and attention values m and a within the range of 0.0 to 100.0. The formula (1) is used to convert the output value m or a into a scaled conversion value denoted as conversion_x:

c o n v e r s i o n_x = (c o n v_m a x - c o n v_m i n) \times \frac{o u t p u t}{(o u t p u t_m a x - o u t p u t_m i n)} - c o n v_m i n

(1)

where x = m or a, output_min and output_max represent the minimum and maximum values of the output values, respectively, and conv_max and conv_min denote the maximum and minimum values of the conversion value conversion_x, respectively. By the formula above, the output values m and a of the EEG device can be proportionally converted into the range of –1.0 to +1.0, with the converted meditation and attention values being denoted as conversion_m and conversion_a, respectively.

Furthermore, let

\vec{O E}

be the vector formed by the point E at the coordinates (conversion_m and conversion_a) and the origin O at (0, 0). Then, by calculating the angle between

\vec{O E}

and the X-axis in an X-Y rectangular coordinate system, one of the 36 angular segments, each spanning 10 degrees, denoted as G, within the 360-degree range that includes the vector

\vec{O E}

can be identified. Finally, according to Algorithm 1, G can be used to identify one of 36 possible emotional states, as illustrated in Figure 19. This figure replicates Figure 6 but with the X- and Y-axes renamed to “meditation” and “attention,” respectively, for this study. Specifically, the horizontal axis in Figure 6, originally representing “pleasure,” is now designated as the meditation value m, while the vertical axis, originally representing “activation,” is now the attention value a. It should also be noted that the X-axis and Y-axis in the affect circumplex of Figure 6 have been alternatively considered to represent the parameters “valence” and “arousal,” respectively, in References [52,53].

For example, if E = (0.5, 0.5), the angular direction of the vector

\vec{O E}

relative to the X-axis is 45°. According to Step 3 of Algorithm 1, 45/10 = 4.5 is calculated and rounded down to the integer I = 5. This indicates that the 10-degree angular segment containing

\vec{O E}

is the 5th segment from the X-axis in the counterclockwise direction, which corresponds to the emotional state “enthusiastic” according to the affect circumplex shown in Figure 19.

Algorithm 1: computing the emotion state from the meditation and attention values.

Step 1: Define two vectors:

- Vector1: Created from the coordinate values (conversion_m, conversion_a) with respective to the origin (0, 0) of the X-Y coordinate system.

- Vector2: A fixed reference vector which is taken to be the X-axis.

Step 2: Calculate the angle between Vector1 and Vector2:

- Use a function to determine the angle θ between Vector1 and Vector2 in the counterclockwise direction with θ to fall in the range of 0 ≤ θ ≤ 360^o.

Step 3: Categorize the angle:

- Divide the angle θ by 10 and round down to the nearest whole number I.

- This integer I represents one of 36 possible 10-degree segments.

Step 4: Decide the output emotion state:

- Find the I-th emotion state counted from the X-axis in Figure 19 in the counterclockwise direction as the desired output.

4.3.5. Visual Displays of System Data and Verbal Descriptions of Users’ Emotion States

As mentioned previously, the Google Cloud API service is used in this study for the analysis of the text data of the user’s speech, resulting in the sentiment-related values called score and magnitude denoted by S and M, respectively. And the Mind Sensor EEG used in this study for the analysis of the user’s brainwaves yields the emotion-related parameter values of attention and meditation, denoted as a and m.

For the purpose of illustrating these data on the interface shown on the TV screen for the user to inspect as shown in Figure 20 which is a copy of the illustration of Step 4.1 in Table 8, the values of these four kinds of parameter data are converted to be in the range of 0.0~1.0, or equivalently, in the range of 0%~100%. Also, four colored scales are employed to indicate the percentages, respectively, with the respective colors being taken to be red, gray, green, and blue, as described in Table 12. Finally, letter papers are also used to show the emotion state of the user, which are of the same colors of the percentage scales.

To illustrate these data on the TV screen interface, as depicted in Figure 20 (a reproduction from Step 4.1 in Table 8), the values of the four types of parameter data are normalized to a range of 0.0 to 1.0, or equivalently, 0% to 100%. Four colored scales—red, gray, green, and blue—are used to represent these percentages, as detailed in Table 12. Additionally, letter papers displaying the user’s emotional state are presented in the same colors as the percentage scales.

Furthermore, the names of the four types of parameter data, namely, score, magnitude, meditation, and attention are renamed as emotion, sensibility, relaxation, and focusing, respectively, for the user to understand more easily.

Finally, both the result yielded by the speech analysis using the Google Cloud NLP API and that yielded by the brainwave analysis using the Mind Sensor EEG are combined to generate a sentence to give a verbal description of the overall emotion state of the user (appearing on top of the four scales in Figure 20), accompanied with a bottle filled with a letter with the color of the emotion scale (“red” for the example shown in Figure 20).

5. System Experience Experiment and Experimental Data Analysis

5.1. System Exhibition and Experiment Processes

An affective computing system called “Emotional Drifting,” has been constructed in this study using multi-sensing interactive interfacing techniques. The system has been publicly showcased in February 2024 with experiments being conducted according to the following procedure.

(1): Experiment Venue: Design Building II, First Floor, and Interactive Multimedia Design Laboratory in Yunlin University of Science and Technology in Yunlin, Taiwan.
(2): Participants: Users aged 18 and above.
(3): Sample Size: 60 participants.
(4): Procedure: The experiment lasts about 20 minutes per participant. Upon arrival, each participant receives a brief overview of the procedure. They then engage with the interactive system while their behaviors are observed. Finally, participants complete a questionnaire survey. The details from a user’s view are shown in Figure 21.

In this section, the experimental process is documented and explained, with direct observation, questionnaire surveys, and brainwave data (attention and meditation) used as the basis for making post-experience adjustments to the system.

5.2. Indirect Observation and Analysis of Collected Data

5.2.1. Ideas of Applying the Indirect Observation Method

In this study, semantic analysis values and EEG data (attention and meditation scores) were collected from participants during their interactions when using the proposed system. Pearson correlation analysis was used to examine the relationships between the data collected from 60 users, focusing on the distribution of correlations and the linear trends observed. A two-tailed test was conducted to explore associations between the data. Correlations were classified into three levels: low, moderate, and high, as detailed in Table 13.

In addition to examining data relations using Pearson correlation tests, significance levels (p-values) are also computed. A correlation coefficient c with a p-value of 0.05 or less is marked with one asterisk (*) as c* and labeled as “significant” (p < 0.05). When the p-value is 0.01 or less, c is denoted with two asterisks (**) as c** and labeled as “highly significant” (p < 0.01). Finally, when the p-value is 0.001 or less, c is marked with three asterisks (***) as c*** and labeled as “very significant” (p < 0.001). These decision thresholds are detailed in Table 14.

Correlation analysis helps in understanding and explaining the relationships between variables, identifying collinearity, and creating visual linear statistical charts. Some applications of this method in this study are investigated subsequently.

5.2.2. Statistical Analysis of Correlations of Semantic and Brainwave Data

In this section, the emotional data collected by the proposed system, “Emotion Drift,” are examined to explore correlations between different users’ emotional states. Such emotion states are expressed either by the semantic content contained in the users’ messages or by the brainwave data (meditation and attention values) measured of the users, to understand their interrelationships. Such correlation analysis includes investigating the transmission of emotions in message exchanges during interpersonal interactions and the impact of previous users’ emotions on current users.

Note that the semantic content of messages is analyzed by use of the Google Speech API, and the brainwave data are obtained by use of the EEG set, both of the proposed system as mentioned previously. The semantic values range from -1.0 to 1.0, while the brainwave data, including meditation and attention values, range from 0 to 100. Such data from the 60 users invited to use the proposed system were analyzed and visualized in detail.

(A): Terminology definitions ---

For convenience of reference, the following terminologies are defined for use in the following discussions. Note that the terminologies are defined from the viewpoint of the current user.

(1): semantic (listening i) – the semantic data of the i-th message left by the previous user which is listened to by the current user, with i = 1 through 4;
(2): semantic (speaking i) – the semantic data of the i-th message spoken by the current user, with i = 1 through 4;
(3): meditation (listening i) – the meditation value measured of the current user when listening to the i-th message left by the previous user;
(4): attention (listening i) – the same as above except that the parameter value is attention;
(5): meditation (speaking i) – the meditation value measured of the current user when speaking the i-th message;
(6): attention (speaking i) – the same as above except that the parameter value is attention;

(B): Types of investigated correlations of emotion states ---

The following types of correlations of emotion data are explored in this study.

(1): Type 1: the correlation between the semantic of the i-th message left by the previous user (and listened by the current user) and the semantic of the j-th message spoken by the current user, i.e., the correlation between semantic (listening i) and semantic (speaking j), i ≠ j.
(2): Type 2: the correlation between the semantic of the i-th message left by the previous user and the meditation or attention value measured of the current user when listening to the j-th message, i.e., the correlation between semantic (listening i) and meditation (speaking j) or attention (speaking j), i ≠ j.
(3): Type 3: the correlation between the meditation (or attention) values measured of the current user while listening to the i-th messages and the meditation (or attention) value while listening to the (i+1)-th message, i.e., the correlation between meditation (listening i) (or attention (listening i)) and meditation (listening i+1) (or attention (listening i+1))
(4): Type 4: the correlation between the meditation (or attention) values measured of the current user while speaking the i-th messages and the meditation (or attention) value while speaking the (i+1)-th message, i.e., the correlation between meditation (speaking i) (or attention (speaking i)) and meditation (speaking i+1) (or attention (speaking i+1)).

(C): An illustrated case of emotion correlation ---

As an illustration of how cases of the previously-mentioned types of correlation of emotion states are investigated, the correlation of semantic (listening 2) and semantic (speaking 4), that is, the correlation between the semantic of the i-th message left by the previous user (and listened to by the current user) and the semantic of the j-th message spoken by the current user, is computed using the 60 users’ emotion data left on the proposed system during the system exhibition period.

The Pearson correlation coefficient is 0.266*, with a significance value of 0.040, as shown in Table 15. The average semantic score for the second message shared by the previous users is 0.30 with a standard deviation of 0.36, while the average score for the fourth message shared by the current users is 0.44 with a standard deviation of 0.39, as detailed in Table 16. The linear distribution of these values is illustrated in Figure 22. Despite the overall scattered linear distribution indicating no significant linear correlation, a notable low-level correlation exists between the data. This suggests that more positive semantic analysis scores in the previous user’s second message are associated with more positive semantic feedback in the current user’s fourth message.

(D): List of the four types of correlations investigated in this study ---

Totally, fifteen cases of the previously-mentioned four types of correlations have been investigated, with the case dealt with above in (C) as one of them. The results are listed in Table 17.

5.2.3. Concluding Remarks of Statistical Analysis

From the results shown in Table 17, the following facts can be drawn.

(1): From all the types:

From the cases presented in the table, it is evident that Emotion dataset 1 is positively correlated with Emotion dataset 2, at least at a statistically significant level (p-value < 0.05).

(2): From Type 1:

Case 1 shows that the more positive the semantic value of the second message shared by the previous user, the more likely the current user is to provide a more positive semantic response when sharing the fourth message.

(3): From Type 2:

From cases 2 to 4, it is observed that a more positive semantic value expressed by the previous user in the left message is likely to influence the current user to exhibit a more positive emotional trend, with increased meditation or attention in their response.

(4): From Type 3:

From cases 5 to 9, it is observed that when a user’s brainwave meditation (or attention) value is high while listening to a message, there is a positive correlation trend, with the user’s brainwave meditation (or attention) value remaining high when listening to the subsequent message.

(5): From Type 4:

From cases 10 to 15, it is observed that when a user’s brainwave meditation (or attention) value is high while sharing a message, there is a positive correlation trend, with the user’s brainwave meditation (or attention) value remaining high when sharing the subsequent message.

5.2.4. Observations of Emotion Changes in Listening and Sharing

In this study, an objective was to investigate changes in the user’s emotions during interactions on the proposed system. As depicted in Figure 19, the two axes of the affect circumplex have been renamed to “meditation” and “attention,” representing relaxation and focusing, respectively, in this study. These axes divide the “emotion space” into four quadrants — enthusiasm, anxiety, depression, and comfort — arranged in a counterclockwise direction.

(A): Message listening phases

Analysis of the aforementioned emotion changes during the interaction process reveals that, in the message-listening phases (I) and (II), most users were situated within the enthusiasm quadrant. However, in phases (III) and (IV), users shifted to the comfort quadrant, as shown in Table 18. This transition indicates that, as the phases progressed, users’ emotions evolved from enthusiasm to a more comfortable state. This suggests that using the proposed system to listen to messages from the previous users during interpersonal interactions can elicit both focusing and relaxation states (represented by the enthusiasm and comfort quadrants, respectively) in the user.

(B): Message sharing phases

An analysis of the emotional changes during the interaction process reveals that in all four message-sharing phases, most users were within the comfort quadrant, as shown in Table 19. This indicates that using the proposed system for sharing messages in interpersonal interactions tends to induce a state of relaxation (represented by the comfort quadrant) in the user.

5.2.5. Concluding Remarks about Emotion Changes while Using the System

The analysis of emotional changes during message-listening phases (I) and (II) shows that users predominantly experienced enthusiasm. As the interaction progressed into phases (III) and (IV), there was a notable shift to a state of comfort. This transition suggests that the system effectively moves the users from an initial state of enthusiasm to a more relaxed and comfortable state. Thus, the system’s message-listening function can engage the users’ focus initially and then foster relaxation, reflecting its capability to evoke both stimulating and calming emotional responses.

In contrast, during all four message-sharing phases, users consistently remained in the comfort quadrant. This indicates that the system tends to induce a state of relaxation throughout the message-sharing process. Therefore, the system’s message-sharing feature primarily supports a relaxing emotional state, enhancing the users’ comfort during interpersonal interactions.

5.3. Questionnaire Survey and Statisitcal Analysis of Answer Data

5.3.1. Statistics of Participants in the Questionnaire Survey

The basic information of the participants in the questionnaire survey of this study, including gender, age, and experience with online dating apps, was analyzed to show the sample structure. A total of 60 participants with valid data were included, with 63% being female and 37% being male. Among them, 78% had used relevant online dating apps. The statistical summary of the basic information of the research sample is shown in Table 20.

5.3.2. Design of Questionnaire

Based on the questionnaire survey method described previously, the questions for the questionnaire survey conducted in this study are designed according to three dimensions, namely “system usability,” “interactive experience,” and “pleasure experience”, using a five-point Likert scale in which opinions ranging from “strongly disagree,” “disagree,” “normal,” “agree,” to “strongly agree” are assigned scores of 1, 2, 3, 4, and 5, respectively. The questionnaires were distributed to the users for them to fill after they used the proposed system. The collected results are shown statistically in Table 21.

5.3.3. Analysis of Reliability and Validity of Collected Questionnaire Data

Before assessing the system’s effectiveness in terms of three dimensions, “system usability,” “interactive experience,” and “pleasure experience” based on questionnaire responses (Table 21), the reliability and validity of the data (Table 19 and Table 20) must be verified. This verification is performed through a five-step process using IBM SPSS and AMOS, with the steps and results detailed below.

(1): Step 1: Verification of the adequacy of the answer dataset

To assess the adequacy of the collected answer data, the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity [135] were applied in this study. Typically, a KMO measure value exceeding the threshold of 0.50 is considered to pass the KMO test. And a significance value below the threshold of 0.05 is generally accepted to pass the Bartlett’s test. When both tests are successful, the dataset is considered to be adequately related for subsequent structural analysis.

With the datasets of the three dimensions of questions shown in Table 21 as input to the SPSS program, the KMO measure values and Bartlett’s significant values can computed to be as those in Table 22. It is evident from the table that all the KMO measure values exceed 0.50, and all the significant values for Bartlett’s test are below 0.05. Therefore, both tests have been successfully passed for each dimension of questions, indicating that the datasets from Table 21 are suitable for further structural analysis, as undertaken in the subsequent steps of data verification.

(2): Step 2: Finding the latent dimensions of the question datasets

In the structural analysis of the answer data of the questionnaire survey, the major aim is to categorize the questions of each dimension into meaningful subsets, each subset being associated with a latent dimension. The exploratory factor analysis (EFA) utilizing principal component analysis, along with the varimax method with Kaiser normalization, was applied for this purpose using the SPSS package. The outcomes with Table 21 as the input are shown in Tables 23 through 25 for the three dimensions, “system usability,” “interactive experience,” and “pleasure experience,” respectively.

After examining Table 23, it is found that the questions in the first dimension “system usability” fall into two groups, RA1 = {A06, A03, A02, A07} and RA2 = {A04, A05, A01}, aligning with two latent dimensions named “functionality” and “effectiveness” in this study according to the common characteristics of the questions in each group. Likewise, from Table 24 and Table 25, similar reasoning can be carried out to group the questions of the second and third dimensions, respectively, and assign names to the resulting latent dimensions. The results, including that for the first dimension, are shown in Table 26.

(3): Step 3: Verifying the reliability of the answer data

Reliability refers to the consistency of a dataset across multiple repetitions, which can be evaluated using the Cronbach’s α coefficient [136] from the EFA mentioned previously. The reliability of the dataset increases as the Cronbach’s α coefficient approaches 1.0. A dataset is considered reliable when the coefficient exceeds 0.35, and it is judged to have high reliability when the coefficient surpasses 0.70 [137].

With the input data of Table 21, the Cronbach’s α coefficients computed by use of the SPSS for the three dimensions and the seven latent dimensions are detailed in Table 27. It is evident from the table that all Cronbach’s α coefficients exceeds 0.35 and most approach or surpass 0.70, meaning that the collected answer data of both the dimensions and the individual latent dimensions are reliable for further analysis.

(4): Step 4: Verification of the applicability of the structural model

To establish the validity of the collected answer data, it is necessary to verify at first the appropriateness of the structural model constructed based on the latent dimensions [138]. This can be done by use of the confirmatory factor analysis (CFA) process, utilizing the AMOS package. The outcome of this analysis conducted in this study with the data of Table 28 as the input resulted in three structure-model graphs, as depicted in Figure 23.

It can be observed that the three index values—χ²/df, cfi, and RMSEA—obtained for each dimension are close to their respective threshold ranges: “between 1 and 5,” “greater than 0.9,” and “between 0.05 and 0.08.” This suggests that the structural model, based on the latent dimensions, fits the collected questionnaire data reasonably well, as indicated by Hu and Bentler [139].

(5): Step 5: Verification of the validity of the question answer data

In Figure 23, which shows the structural model of the questionnaire data, all factor loading values (standardized regression weights) for the paths along the dimensions to the question items exceed or are very close to the threshold of 0.5. This observation suggests that the construct validity of the model is quite good. This validation is further supported by the construct validity values of all latent dimensions calculated through the EFA process, as detailed in Table 29, where it is seen that each value surpasses or is very close to the threshold of 0.6 for confirming construct validity [140].

Through the above five steps of statistical processes, the verification of the reliability and validity of the questionnaire data is completed. Then, it can be proceeded to analyze the questionnaire data of each latent dimension, as conducted in the subsequent discussions.

5.3.4. Detailed Analyses of Answer Data of the Dimension of System Usability

(A): Analysis about the latent dimension of “functionality”

The primary purpose of the questions related to this latent dimension, “functionality,” is to explore the functional uses of the proposed system “Emotion Drift” as shown in Table 30, which indicates that the proposed system designed in this study has moderate usability.

(1): The average values for this latent dimension range from 4.00 to 4.43, with standard deviations between 0.62 and 0.86, indicating user agreement with all question items.
(2): The average value for question A06, “the various functions of the system are well integrated,” is 4.43 and the standard deviation is 0.59, with over 95% agreement, showing strong consensus on the effectiveness of function integration.
(3): The average value for question A02, “I am able to understand and become familiar with each function of the system,” is 4.23 and the standard deviation is 0.62, with over 90% agreement, suggesting the system is easy for most users to understand.
(4): The average value for question A07, “ I would like to use this system frequently,” is 4.00 and the standard deviation is 0.78, with over 80% agreement, but over 15% of the users were neutral, indicating some indifference regarding the willingness to use the system again.
(5): All four items in this latent dimension have over 80% agreement, reflecting that the users find the proposed system to have good usability.

(B): Analysis about the latent dimension of “effectiveness”

The primary purpose of the questions related to this latent dimension, “effectiveness,” is to explore the users’ coherent experience of emotional perception and system processes during interaction as shown in Table 31, which indicates that the users of the proposed system have effective coherent experience in using the system.

(1): The average values for this dimension range from 4.00 to 4.38, with standard deviations between 0.61 and 0.75, indicating high perceived effectiveness of the system.
(2): The standard deviation value for question A04, “the emotions output by the system seem accurate,” is above 0.7, showing some variation in user opinions. However, with an average of 4.0, it indicates that most users agree that their perceived emotions align with the system’s results.
(3): Excluding question A04, each remaining question has over 90% agreement and a standard deviation below 0.62, reflecting a consistent user understanding of the effectiveness of the proposed system.

5.3.5. Detailed Analyses of Answer Data of the Dimension of Interactive Experience

(A): Analysis about the latent dimension of “empathy”

The aim of the questions related to this latent dimension, “empathy,” is to explore the extent to which users believe the use of the proposed system “Emotion Drift” fosters empathy-related emotions as shown in Table 32.

(1): The average values for this latent dimension range from 3.9 to 4.01, with standard deviations between 0.80 and 1.13, indicating that most users have a positive experience regarding the motions related to empathy during the interactions.
(2): For questions B01, B02, and B04, more than 15% of responses were neutral, indicating notable variability in user opinions. Despite these differences, the overall response still reflects a positive user experience.
(3): Especially, questions B04 and B01, both about empathy and care about others, have respectively 77% and 79% of agreements or above, meaning that most of the users improved their feelings of these aspects after using the system.

(B): Analysis about the latent dimension of “isolation”

The extent of negative feelings experienced by users while using the system is primarily explored in this latent dimension. All question items in this dimension are reverse-coded, and the results are derived by reversing the scores of the negative items in the questionnaire, as shown in Table 33.

(1): The average values for this dimension range from 3.2 to 4.06, with standard deviations between 0.93 and 1.29, indicating that a positive tendency in user interactions with the system is generally experienced, though with a few neutral responses.
(2): There are three question items in this latent dimension, all with standard deviations greater than 0.9, and questions B03 and B07 have standard deviations exceeding 1.0, indicating considerable divergence in user opinions for these questions.
(3): For question B05, “I feel lonely while using the system,” an agreement rate of over 80% was observed, suggesting that the system helps alleviate loneliness and enhances interpersonal relationships.

5.3.6. Detailed Analyses of Answer Data of the Dimension of Pleasure Experience

(A): Analysis about the latent dimension of “satisfaction”

The extent to which users experience pleasure and find the proposed system cognitively interesting is primarily explored in this latent dimension. The results of the analysis, presented in Table 34, indicate that good pleasure-inducing qualities are exhibited by the proposed system, which satisfy the users.

(1): The average values for this latent dimension range from 4.11 to 4.73, with standard deviations between 0.54 and 0.94, reflecting the users’ good experiences of pleasure while using the proposed system.
(2): For question C04, “I find the system’s interactive experience interesting,” an average value of 4.73 and a standard deviation of 0.54 were recorded, with 99% agreement, indicating that the users are highly satisfied with the interaction process of the system, thinking it to be very interesting and really enjoyable.
(3): For question C09, “time seems to fly when I am using the system,” the average was 4.11, with a standard deviation of 0.94 and 18% of neutral responses. This indicates that there is discrepancy in user perceptions and thoughts regarding this question.
(4): Aside from question C09, questions C04 and C05 achieved over 86% agreements, suggesting that most users perceive the system as being able to provide positive emotional experiences like happiness and pleasure.

(B): Analysis about the latent dimension of “immersion”

The extent of immersion experienced by the users with the proposed system is primarily explored in this latent dimension. The results of the analysis, presented in Table 35, indicate that moderate pleasure-inducing qualities are exhibited by the proposed system, which comes from the users’ emersion in using the system.

(1): The average values for this latent dimension range from 4.23 to 4.45, with the standard deviations between 0.67 and 0.88, indicating that positive and pleasant experiences are generally reported by the users.
(2): For question C07, “I enjoy using the system,” an average value of 4.45 and a standard deviation of 0.67 were observed, with over 93% agreement, indicating that high affirmation of the pleasurable experience provided by the system is given by users.
(3): For question C09, “time seems to fly when I am using the system,” the average is 4.11, with a standard deviation of 0.94 and 18% of neutral responses. This indicates that some discrepancy exists in user perceptions and thoughts regarding this question.
(4): This latent dimension includes three questions, with all achieving over 84% agreement, suggesting that the proposed system is generally perceived by the users as providing enjoyable, immersive, and satisfying emotional experiences throughout the interaction process.

(C): Analysis about the latent dimension of “relaxation”

The extent of immersion experienced by the users with the proposed system is primarily explored in this latent dimension. The results of the analysis, presented in Table 36, indicate that moderate pleasure-inducing qualities are exhibited by the proposed system, which comes from the users’ relaxation while using the system.

(1): The average values for this latent dimension range from 3.85 to 4.36, with standard deviations between 0.66 and 0.84, indicating that a positive and active experience regarding stress relief is reported by the users during the interactions.
(2): Among the three questions in this latent dimension, question C01, “I feel special when using the system,” has a standard deviation of 0.84 with 28% of users providing neutral responses, the highest proportion in this latent dimension. This indicates noticeable variation in users’ opinions regarding this question.
(3): Questions C02 and C03 have over 85% agreements, indicating that the proposed system is perceived by users as highly effective in providing relaxation and stress relief.

5.3.7. Concluding Remarks about the Analysis of the Questionnaire Survey Results

From the previously presented statistical analysis of the questionnaire survey results, the following conclusions can be drawn.

(1): In the system usability dimension of the questionnaire, which is divided into two latent dimensions — “functionality” and “effectiveness” — the overall analysis indicates that the Emotion Drift System designed in this study demonstrates moderate usability, as reported by users after interacting with the system.
(2): In the interactive experience dimension of the questionnaire, which includes the latent dimensions of “empathy” and “isolation,” questions B03, B06, and B07 have standard deviations greater than 1.0, reflecting variability in users’ responses. Nevertheless, the overall results show that the system effectively reduces loneliness, fosters empathy, and enhances interpersonal interactions.
(3): In the pleasure experience dimension, which is divided into three latent dimensions — “satisfaction,” “immersion,” and “relaxation” — the overall analysis reveals that the Emotion Drift System demonstrates satisfactory pleasure-inducing qualities.

6. Conclusions and Suggestions

In this section, conclusions based on the findings of this study are drawn, followed by some suggestions for future studies are included.

6.1. Conclusions

In this study, the use of multi-sensing interactive interfaces for affective computing to evoke emotional resonance and enhance positive interpersonal interactions has been explored. A comprehensive review of cutting-edge literature and relevant case studies revealed that sharing and exchanging messages in interpersonal interactions conveys emotional information, with pleasure, enjoyment, and interest being crucial for sustaining engagement. The review also highlighted that multimodal interactive interfaces offer flexibility by removing constraints related to form, time, and space, and that systems with emotional functions can recognize human emotions in real-time.

Based on these findings, design principles were developed for creating a system to achieve these goals. A prototype emotion-perception system named “Emotion Drift” was constructed to address negative emotions in social relationships, promote empathetic communication, and provide a higher-quality, more human-centered experience to strengthen interpersonal interactions.

The system was tested through experiments involving 60 participants. Indirect observation and questionnaires were used to assess its effectiveness, and the collected data were statistically analyzed to draw the following conclusions.

(1): Emotional transmission in interactions is confirmed, with overall positive user emotions and notable individual variations ?

Emotional transmission in interpersonal interactions is confirmed, with the proposed emotion-perception system “Emotion Drift” identifying 36 distinct emotions. Analysis showed that while users’ overall emotions tended to be positive, there were notable variations in individual emotional experiences.

(2): The proposed emotional perception system “Emotion Drift” effectively stimulates positive emotions in users and strengthens interpersonal interactions ?

The incorporation of affective computing technology into interpersonal experiences validated that the system promotes positive, enjoyable emotional resonance among users, leading to high-quality interpersonal relationships.

(3): A positive correlation was found to exist between semantic analysis and brainwave metrics of relaxation and concentration ?

Specifically, during the message interaction process, users exhibited relaxed emotions. However, during topic transition to the third message, both the listening and sharing processes effectively increased the user’s concentration and enhanced the quality of positive emotional interaction.

(4): The interaction experience based on multimodal and affective computing provides users with pleasure and relaxation emotions ?

Analysis of emotional trends revealed that during listening, brainwave emotions were mainly in the Enthusiasm or Comfort quadrants, while sharing focused more on Comfort. The data confirm a shift from enthusiasm to comfort, demonstrating that the “Emotion Drift” system offers a pleasant and relaxing emotional experience.

6.2. Suggestions for Future Research

Due to the limitations of this study, there are several areas for improvement in the system design. The following directions are suggested for further research:

(1): Exploring multi-sensing interaction and affective computing across different age groups?

This study’s sample mainly comprised individuals aged 18-25. Future research should explore emotional transmission trends across different age groups and examine individual users’ emotional data, which this study did not address.

(2): Choosing a fixed, quiet, comfortable experimental environment ?

To accurately measure physiological emotional data, experiments should be conducted in a stable, serene, and comfortable environment. This setting will help users relax and exhibit natural emotional responses, leading to more reliable emotional data and improved understanding of emotion transmission.

(3): Designing longer induction phases to better cultivate emotions ?

In this study, it was noted that the user’s relaxation feeling increases from the second to the fourth message during the listening phase. To enhance emotional engagement, it is suggested to extend the initial induction phase, giving users more time to fully engage and display emotional trends.

(4): Using more intuitive and readable visualizations for emotional information ?

The current system only allows users to infer others’ emotions through voice and context. Future research should explore incorporating more intuitive and proactive visualizations to actively present emotional information to users.

(5): Conducting experiments with diverse multi-sensing interaction interfaces and affective computing technologies ?

This study used voice and gesture recognition with Google Cloud semantic analysis and a lightweight EEG sensor. Future researchers should explore diverse interfaces and affective technologies to expand the scope of the study.

Author Contributions

Conceptualization, C.-M.W. and Y.-C.L.; methodology, C.-M.W. and Y.-C.L.; validation, Y.-C.L.; investigation, Y.-C.L.; data curation, Y.-C.L.; writing—original draft preparation, C.-M.W.; writing—review and editing, C.-M.W.; visualization, Y.-C.L.; supervision, C.-M.W.; funding acquisition, C.-M.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Informed Consent Statement

All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki.

Data Availability Statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

References

Dubé, J. P.; Smith, M. M.; Sherry, S.B.; Hewitt, P. L.; Stewart, S. H. Suicide behaviors during the COVID-19 pandemic: A meta-analysis of 54 studies. Psychiatry Res. 2021, 301, 113998. [Google Scholar] [CrossRef] [PubMed]
Krogstad, J.; Passel, J.; Cohn, D.V. Pew Research Center. US Border Apprehensions Of Families And Unaccompanied Children Jump Dramatically.
World Mental Health Day. World Health Organization. Available online: https://www.who.int/campaigns/world-mental-health-day/2023 (accessed on 22 August 2024).
Arya, R.; Singh, J.; Kumar, A. A survey of multidisciplinary domains contributing to affective computing. Comput. Sci. Rev. 2021, 40, Article 100399. [CrossRef]
Collins, R. Interaction ritual chains. In Interaction Ritual Chains; Princeton University Press: Princeton, NJ, USA, 2014. [Google Scholar]
Meng, L.; Zhao, Y.; Jiang, Y.; Bie, Y.; Li, J. Understanding interaction rituals: The impact of interaction ritual chains of the live broadcast on people’s wellbeing. Front. Psychol. 2022, 13. [Google Scholar] [CrossRef] [PubMed]
Zhao, S.; Wang, S.; Soleymani, M.; Joshi, D.; Ji, Q. Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Trans. Multim. Comput. Commun. Appl. (TOMM) 2019, 15, 1–32. [Google Scholar] [CrossRef]
Picard, R.W. Affective computing: from laughter to IEEE. IEEE Trans. Affect. Comput. 2010, 1, 11–17. [Google Scholar] [CrossRef]
Samadiani, N.; Huang, G.; Cai, B.; Luo, W.; Chi, C.-H.; Xiang, Y.; He, J. A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors 2019, 19, 1863. [Google Scholar] [CrossRef]
Gervasi, R.; Barravecchia, F.; Mastrogiacomo, L.; Franceschini, F. Applications of affective computing in human-robot interaction: State-of-art and challenges for manufacturing. Proc. Inst. Mech. Eng. Part B-J. Eng. Manuf. 2022. [Google Scholar] [CrossRef]
Yongda, D.; Fang, L.; Huang, X. Research on multimodal human-robot interaction based on speech and gesture. Comput. Electr. Eng. 2018, 72, 443–454. [Google Scholar] [CrossRef]
Samaroudi, M.; Echavarria, K.R.; Perry, L. Heritage in lockdown: digital provision of memory institutions in the UK and US of America during the COVID-19 pandemic. Museum Manag. Curatorship 2020, 35, 337–361. [Google Scholar] [CrossRef]
Coherent Market Insights. Affective Computing Market Analysis. Available online: https://www.coherentmarketinsights.com/market-insight/affective-computing-market-5069 (accessed on 22 August 2024).
DeVito, J.A. The Interpersonal Communication Book, 13th ed.; United: New York, NY, USA, 2013. [Google Scholar]
Manning, J. Interpersonal communication. In The SAGE International Encyclopedia of Mass Media and Society; SAGE Publications: Thousand Oaks, CA, USA, 2020; Volume 2, pp. 842–845. [Google Scholar]
Burgoon, J.K.; Berger, C.R.; Waldron, V. R. Mindfulness and interpersonal communication. J. Soc. Issues 2000, 56, 105–127. [Google Scholar] [CrossRef]
Norman, D. The Design of Everyday Things: Revised and Expanded Edition; Basic Books: New York, NY, USA, 2013. [Google Scholar]
Auxier, B.; Anderson, M. Social media use in 2021. Pew Res. Center 2021, 1, 1–4. [Google Scholar]
Derks, D.; Fischer, A.H.; Bos, A.E. The role of emotion in computer-mediated communication: A review. Comput. Hum. Behav. 2008, 24, 766–785. [Google Scholar] [CrossRef]
Derks, D.; Bos, A.E.; Von Grumbkow, J. Emoticons and social interaction on the Internet: The importance of social context. Comput. Hum. Behav. 2007, 23, 842–849. [Google Scholar] [CrossRef]
Li, C.; Ning, G.; Xia, Y.; Guo, K.; Liu, Q. Does the internet bring people closer together or further apart? The impact of internet usage on interpersonal communications. Behav. Sci. 2022, 12, 425. [Google Scholar] [CrossRef] [PubMed]
Chartrand, T.L.; Bargh, J.A. The chameleon effect: The perception–behavior link and social interaction. J. Pers. Soc. Psychol. 1999, 76, 893. [Google Scholar] [CrossRef]
Reis, H.T.; Regan, A.; Lyubomirsky, S. Interpersonal chemistry: What is it, how does it emerge, and how does it operate? Perspect. Psychol. Sci. 2022, 17, 530–558. [Google Scholar] [CrossRef]
Miles, L.K.; Nind, L.K.; Macrae, C.N. The rhythm of rapport: Interpersonal synchrony and social perception. J. Exp. Soc. Psychol. 2009, 45, 585–589. [Google Scholar] [CrossRef]
Stephens, J.P.; Heaphy, E.; Dutton, J.E. High-quality connections. In Cameron, K. and Spreitzer, G., Eds., Handbook of Positive Organizational Scholarship; Oxford University Press: New York, NY, USA, 2012; pp. 385–399. [Google Scholar]
Jiarui, W.; Xiaoli, Z.; Jiafu, S. Interpersonal relationship, knowledge characteristic, and knowledge sharing behavior of online community members: A TAM perspective. Comput. Intell. Neurosci. 2022, 2022, 4188480. [Google Scholar] [CrossRef]
Davis, F.D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989, 13, 319–340. [Google Scholar] [CrossRef]
Lu, Y.; Zhou, T.; Wang, B. Exploring Chinese users’ acceptance of instant messaging using the theory of planned behavior, the technology acceptance model, and the flow theory. Comput. Hum. Behav. 2009, 25(1), 29–39. [Google Scholar] [CrossRef]
Venkatesh, V.; Speier, C.; Morris, M.G. User acceptance enablers in individual decision making about technology: Toward an integrated model. Decis. Sci. 2002, 33, 297–316. [Google Scholar] [CrossRef]
Mun, Y.Y.; Hwang, Y. Predicting the use of web-based information systems: Self-efficacy, enjoyment, learning goal orientation, and the technology acceptance model. Int. J. Hum. Comput. Stud. 2003, 59, 431–449. [Google Scholar]
Chung, J.; Tan, F.B. Antecedents of perceived playfulness: An exploratory study on user acceptance of general information-searching websites. Inf. Manag. 2004, 41, 869–881. [Google Scholar] [CrossRef]
Brown, J.S.; Collins, A.; Duguid, P. Situated cognition and the culture of learning. Educ. Res. 1989, 18, 32–42. [Google Scholar] [CrossRef]
Durning, S.; Artino, A. Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52. Med. Teach. 2011, 33, 188–199. [Google Scholar] [CrossRef]
Schrepp, M.; Hinderks, A.; Thomaschewski, J. Design and evaluation of a short version of the User Experience Questionnaire (UEQ-S). Int. J. Interact. Multimedia Artif. Intell. 2017, 4, 103. [Google Scholar] [CrossRef]
Collins, R. Princeton University Press. 2005. [CrossRef]
Bellocchi, A.; Quigley, C.; Otrel-Cass, K. Exploring emotions, aesthetics and wellbeing in science education research, Vol. 13; Springer: Heidelberg, Germany, 2016. [Google Scholar]
Krishnan, R.; Cook, K.S.; Kozhikode, R.K.; Schilke, O. An Interaction Ritual Theory of Social Resource Exchange: Evidence from a Silicon Valley Accelerator. Admin. Sci. Q. 2021, 66, 659-710, Article 0001839220970936. [CrossRef]
Patricia, M. Online networks and emotional energy: How pro-anorexic websites use interaction ritual chains to (re)form identity. Inf. Commun. Soc. 2012, 16, 105–124. [Google Scholar]
Clark, H.H. Using language; Cambridge University Press: Cambridge, UK, 1996. [Google Scholar]
Van Kleef, G.A. How emotions regulate social life: The emotions as social information (EASI) model. Curr. Dir. Psychol. Sci. 2009, 18, 184–188. [Google Scholar] [CrossRef]
Van Kleef, G.A.; De Dreu, C.K.W.; Manstead, A.S.R. An interpersonal approach to emotion in social decision making: The emotions as social information model. In Advances in Experimental Social Psychology, Vol. 42; Academic Press: Cambridge, MA, USA, 2010; pp. 45–96. [Google Scholar] [CrossRef]
Wright, P.; McCarthy, J. Technology as Experience; MIT Press: Cambridge, MA, USA, 2004. [Google Scholar]
Hallnäs, L.; Redström, J. Slow technology–designing for reflection. Pers. Ubiquitous Comput. 2001, 5, 201–212. [Google Scholar] [CrossRef]
Picard, R.W. Affective Computing; The MIT Press: Cambridge, MA, USA, 1997. [Google Scholar]
Breazeal, C.; Scassellati, B. How to build robots that make friends and influence people. In Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE: Piscataway, NJ, USA, 1999; pp. 858-863.
Shukla, A.; Gullapuram, S.S.; Katti, H.; Kankanhalli, M.; Winkler, S.; Subramanian, R. Recognition of advertisement emotions with application to computational advertising. IEEE Trans. Affect. Comput. 2022, 13, 781–792. [Google Scholar] [CrossRef]
Andalibi, N.; Buss, J. The human in emotion recognition on social media: Attitudes, outcomes, risks. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; ACM: New York, NY, USA, 2020; pp. 1–15. [Google Scholar]
Shi, Z. Intelligence Science: Leading the Age of Intelligence; Elsevier: Amsterdam, Netherlands, 2021. [Google Scholar]
Sloman, A. Review of affective computing. AI Mag. 1999, 20, 127–127. [Google Scholar]
Plutchik, R. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am. Sci. 2001, 89, 344–350. [Google Scholar] [CrossRef]
Cambria, E.; Livingstone, A.; Hussain, A. The hourglass of emotions. In Cognitive Behavioural Systems: COST 2102 International Training School, Dresden, Germany, February 21-26, 2011; Revised Selected Papers; Springer: Berlin, Germany, 2012; pp. 112–121. [Google Scholar]
Feldman, L.A. Valence focus and arousal focus: Individual differences in the structure of affective experience. J. Pers. Soc. Psychol. 1995, 69, 153–166. [Google Scholar] [CrossRef]
Russell, J.A. A circumplex model of affect. J. Pers. Soc. Psychol. 1980, 39, 1161–1178. [Google Scholar] [CrossRef]
Russell, J.A. Core affect and the psychological construction of emotion. Psychol. Rev. 2003, 110, 145–172. [Google Scholar] [CrossRef]
Warr, P.; Inceoglu, I. Job engagement, job satisfaction, and contrasting associations with person–job fit. J. Occup. Health Psychol. 2012, 17, 129–148. [Google Scholar] [CrossRef] [PubMed]
Kensinger, E.A. Remembering the details: Effects of emotion. Emotion Rev. 2009, 1, 99–113. [Google Scholar] [CrossRef] [PubMed]
Basu, S.; Jana, N.; Bag, A.; Mahadevappa, M.; Mukherjee, J.; Kumar, S.; Guha, R. Emotion recognition based on physiological signals using valence-arousal model. In 2015 Third International Conference on Image Information Processing (ICIIP); IEEE: Piscataway, NJ, USA, 2015; pp. 566–571. [Google Scholar]
Jefferies, L.N.; Smilek, D.; Eich, E.; Enns, J.T. Emotional valence and arousal interact in attentional control. Psychol. Sci. 2008, 19, 290–295. [Google Scholar] [CrossRef]
Nicolaou, M.A.; Gunes, H.; Pantic, M. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. Affect. Comput. 2011, 2, 92–105. [Google Scholar] [CrossRef]
Ramirez, R.; Vamvakousis, Z. Detecting emotion from EEG signals using the Emotiv Epoc device. In Brain Informatics: International Conference, BI 2012, Macau, China, December 4-7, 2012; Proceedings; Springer: Berlin, Germany, 2012; pp. 125–134. [Google Scholar]
Wang, C.-M.; Chen, Y.-C. Design of an interactive mind calligraphy system by affective computing and visualization techniques for real-time reflections of the writer’s emotions. Sensors 2020, 20(20), 5741. [Google Scholar] [CrossRef]
da Silva, F.L. Neural mechanisms underlying brain waves: from neural membranes to networks. Electroencephalography and Clinical Neurophysiology 1991, 79, 81–93. [Google Scholar] [CrossRef]
Shackman, A.J.; McMenamin, B.W.; Maxwell, J.S.; Greischar, L.L.; Davidson, R.J. Identifying robust and sensitive frequency bands for interrogating neural oscillations. Neuroimage 2010, 51, 1319–1333. [Google Scholar] [CrossRef] [PubMed]
Yudhana, A.; Mukhopadhyay, S.; Karas, I.R.; Azhari, A.; Mardhia, M.M.; Akbar, S. A.; Muslim, A.; Ammatulloh, F. I. Recognizing human emotion patterns by applying Fast Fourier Transform based on brainwave features. In 2019 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) 2019.
Sundström, P.; Ståhl, A.; Höök, K. In situ informants exploring an emotional mobile messaging system in their everyday practice. International Journal of Human-Computer Studies 2007, 65, 704–716. [Google Scholar]
Ko, L.-W.; Yang, C.-S. How emotion processing influences post-purchase intention: An exploratory study of consumer electronic products. International Journal of Information Management 2013, 33, 731–740. [Google Scholar]
Sung, J.-Y.; Kim, J.-W.; Kim, J.-H. Cognitive and emotional responses to an interactive product: The case of the iPod nano. International Journal of Human-Computer Studies 2009, 67, 610–622. [Google Scholar]
D’Mello, S. K.; Graesser, A. C. Multimodal sensors for assessing affective states. In Affective Computing and Intelligent Interaction; Springer: Berlin, Germany, 2012; pp. 54–64. [Google Scholar]
Fitzpatrick, K.K.; Darcy, A.; Vierhile, M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Mental Health 2017, 4, e7785. [Google Scholar] [CrossRef] [PubMed]
Tseng, Y.C. The Proof of existence and strategy of performativity - My name is Jane. Artistica TNNUA 2021, 23, 1-17 (in Chinese).
Boehner, K.; DePaula, R.; Dourish, P.; Sengers, P. How emotion is made and measured. International Journal of Human-Computer Studies 2007, 65, 275-291. [CrossRef]
Gartner. 5 Trends Appear on the Gartner Hype Cycle for Emerging Technologies. Available online: https://www.gartner.com/smarterwithgartner/5-trends-appear-on-the-gartner-hype-cycle-for-emerging-technologies-2019 (accessed on 24 August 2024).
Cambria, E.; Das, D.; Bandyopadhyay, S.; Feraco, A. Affective computing and sentiment analysis. In A Practical Guide to Sentiment Analysis; Springer: Berlin, Germany, 2017; pp. 1–10. [Google Scholar]
Poria, S.; Cambria, E.; Bajpai, R.; Hussain, A. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion 2017, 37, 98–125. [Google Scholar] [CrossRef]
Fang, Y.; Hettiarachchi, N.; Zhou, D.; Liu, H. Multi-modal sensing techniques for interfacing hand prostheses: A review. In 2015 IEEE Sensors Journal 2015; pp. 6065-6076.
Wilhelm, M.; Krakowczyk, D.; Albayrak, S. PeriSense: ring-based multi-finger gesture interaction utilizing capacitive proximity sensing. Sensors 2020, 20(14), 3990. [Google Scholar] [CrossRef]
Due, B. L.; Licoppe, C. Video-mediated interaction (VMI): introduction to a special issue on the multimodal accomplishment of VMI institutional activities. Social Interaction. Video-Based Studies of Human Sociality 2021, 3(3).
Ferscha, A.; Resmerita, S.; Holzmann, C.; Reichör, M. Orientation sensing for gesture-based interaction with smart artifacts. Computer Communications 2005, 28, 1552–1563. [Google Scholar] [CrossRef]
Jonell, P.; Bystedt, M.; Fallgren, P.; Kontogiorgos, D.; Lopes, J.; Malisz, Z.; Mascarenhas, S.; Oertel, C.; Raveh, E.; Shore, T. Farmi: a framework for recording multi-modal interactions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) 2018.
Ueng, S.-K. Vision based multi-user human computer interaction. Ph.D. Thesis, National Taiwan Normal University, Taipei, Taiwan, 2016. [Google Scholar]
Yasen, K.N. Vision-based control by hand-directional gestures converting to voice. International Journal of Scientific & Technology Research 2018, 7, 185–190. [Google Scholar]
Bolt, R.A. “Put-that-there” Voice and gesture at the graphics interface. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques 1980.
Billinghurst, M. Put that where? Voice and gesture at the graphics interface. ACM Siggraph Computer Graphics 1998, 32, 60–63. [Google Scholar] [CrossRef]
Jalil, N. Introduction to Intelligent User Interfaces (IUIs). In Software Usability; IntechOpen: London, UK, 2021. [Google Scholar]
Song, J.-H.; Min, S.-H.; Kim, S.-G.; Cho, Y.; Ahn, S.-H. Multi-functionalization strategies using nanomaterials: A review and case study in sensing applications. Int’l J. of Precision Engineering & Manufacturing-Green Technology 2022, 9, 323–347. [Google Scholar]
Maybury, M.; Wahlster, W. Readings in Intelligent User Interfaces; Morgan Kaufmann: San Francisco, CA, 1998. [Google Scholar]
Škraba, A.; Koložvari, A.; Kofjač, D.; Stojanović, R. Wheelchair maneuvering using leap motion controller and cloud based speech control: Prototype realization. 2015 4th Mediterranean Conference on Embedded Computing (MECO) 2015.
Rodríguez-Hidalgo, C.; Pantoja, A.; Araya, F.; Araya, H.; Dias, V. Generating empathic responses from a social robot: An integrative multimodal communication framework using Sima Robot. Workshop on Behavioral Patterns and Interaction Modelling for Personalized Human-Robot Interaction, Cambridge, UK, March 23, 2020.
Park, C.M.; Ki, T.; Ben Ali, A. J.; Pawar, N.S.; Dantu, K.; Ko, S.Y.; Ziarek, L. Gesto: Mapping UI events to gestures and voice commands. Proceedings of the ACM on Human-Computer Interaction 2019, 3, 1-22.
Lin, H.C.; Tseng, Y.C.; Chang, C.C. Affective Computing - Creative applications of emotional analysis. Science Monthly 2020, 569, 55-63 (in Chinese).
Kaczmarek, W.; Panasiuk, J.; Borys, S.; Banach, P. Industrial robot control by means of gestures and voice commands in off-line and on-line mode. Sensors 2020, 20, 6358. [Google Scholar] [CrossRef]
Scupin, R. The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology. Human Organization 1997, 56, 2. [Google Scholar] [CrossRef]
Cohen, E.; Aberle, D.F.; Bartolomé, L.J.; Caldwell, L.K.; Esser, A.H.; Hardesty, D. L.; Hassan, R.; Heinen, H. D.; Kawakita, J.; Linares, O.F. Environmental Orientations: A Multidimensional Approach to Social Ecology [and Comments and Reply]. Current Anthropology 1976, 17, 49–70. [Google Scholar] [CrossRef]
Kuniavsky, M. Observing the User Experience: A Practitioner’s Guide to User Research 2003.
Beyer, H.; Holtzblatt, K. Contextual design. Interactions 1999, 6, 32–42. [Google Scholar] [CrossRef]
Holtzblatt, K.; Wendell, J.; Wood, S. Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design; Morgan Kaufmann: San Francisco, CA, 2004; p. 320. [Google Scholar]
Awasthi, A.; Chauhan, S.S. A hybrid approach integrating Affinity Diagram, AHP and fuzzy TOPSIS for sustainable city logistics planning. Applied Mathematical Modelling 2012, 36, 573–584. [Google Scholar] [CrossRef]
Lucero, A. Using affinity diagrams to evaluate interactive prototypes. In Human-Computer Interaction–INTERACT 2015: 15th IFIP TC 13 International Conference, Bamberg, Germany, September 14-18, 2015, Proceedings, Part II.
Hanington, B.; Martin, B. Universal Methods of Design Expanded and Revised: 125 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions; Rockport Publishers: Beverly, MA, 2019. [Google Scholar]
Cooper, A. The Inmates Are Running the Asylum; Springer: New York, NY, 1999. [Google Scholar]
Cooper, A.; Reimann, R. About Face 2.0: The Essentials of Interaction Design; Wiley: New York, NY, 2003. [Google Scholar]
Acuña, S.T.; Castro, J.W.; Juristo, N. A HCI technique for improving requirements elicitation. Information and Software Technology 2012, 54, 1357–1375. [Google Scholar] [CrossRef]
Pileggi, S.F. Knowledge interoperability and re-use in Empathy Mapping: An ontological approach. Expert Systems with Applications 2021, 180, 115065. [Google Scholar] [CrossRef]
Ferreira, B.; Silva, W.; Oliveira, E.; Conte, T. Designing Personas with Empathy Map. SEKE 2015. [Google Scholar]
Kalbach, J. Maximize business impact with JTBD, empathy maps, and personas. UX Design. Available online: https://www.uxdesign.cc (accessed on 24 August 2024).
Kasemsarn, K. A Marketing Plan for Applying 5W1H: A Case Study from Thailand. The International Journal of Design Education 2022, 16, 37. [Google Scholar] [CrossRef]
Cooper, A.; Reimann, R.; Cronin, D. About Face 3: The Essentials of Interaction Design; John Wiley & Sons: Hoboken, NJ, 2007. [Google Scholar]
Govella, A. Collaborative Product Design: Help Any Team Build a Better Experience; O’Reilly Media: Sebastopol, CA, 2019. [Google Scholar]
Kalbach, J. Mapping Experiences; O’Reilly Media: Sebastopol, CA, 2020. [Google Scholar]
Pierdicca, R.; Paolanti, M.; Quattrini, R.; Mameli, M.; Frontoni, E. A Visual Attentive Model for Discovering Patterns in Eye-Tracking Data-A Proposal in Cultural Heritage. Sensors 2020, 20, 2101. [Google Scholar] [CrossRef]
Ulwick, A.W.; Osterwalder, A. Jobs to be Done: Theory to Practice 2016.
Hornbæk, K.; Stage, J. The interplay between usability evaluation and user interaction design. International Journal of Human-Computer Interaction 2006, 21, 117-123.
Bernhaupt, R.; Palanque, P.; Manciet, F.; Martinie, C. User-test results injection into task-based design process for the assessment and improvement of both usability and user experience. In Proceedings of 6th International Conference on Human-Centered Software Engineering, HCSE 2016, and 8th International Conference on Human Error, Safety, and System Development, HESSD 2016, Stockholm, Sweden, August 29-31, 2016.
Bernard, H. Boar, Application Prototyping: A requirements definition strategy for the 80s. In New York: John Wiley & Sons 1984.
Schork, S.; Kirchner, E. Defining requirements in prototyping: The holistic prototype and process development. DS 91: Proceedings of NordDesign 2018, Linköping, Sweden, 14th-17th August 2018 2018.
Turner, J.H. The origins of positivism: The contributions of Auguste Comte and Herbert Spencer. In Handbook of Social Theory; Sage: Thousand Oaks, CA, 2001; pp. 30–42. [Google Scholar]
Williamson, K. Research Methods for Students, Academics and Professionals: Information Management and Systems; Elsevier: Amsterdam, Netherlands, 2002. [Google Scholar]
Baker, L. Observation: A complex research method. Library Trends 2006, 55, 171–189. [Google Scholar] [CrossRef]
Radzikowski, J.; Stefanidis, A.; Jacobsen, K.H.; Croitoru, A.; Crooks, A.; Delamater, P. L. The measles vaccination narrative in Twitter: A quantitative analysis. JMIR Public Health and Surveillance 2016, 2, e5059. [Google Scholar] [CrossRef]
Taherdoost, H. Validity and reliability of the research instrument; how to test the validation of a questionnaire/survey in a research. Available online: https://doi.org/10.2139/ssrn.2815600 (accessed on 10 May 2023).
Krosnick, J.A. Questionnaire design. In the Palgrave Handbook of Survey Research; Palgrave Macmillan: London, UK, 2018; pp. 439–455. [Google Scholar]
Brooke, J. SUS: a “quick and dirty” usability. In Usability Evaluation in Industry; Taylor & Francis: London, UK, 1996; pp. 189–194. [Google Scholar]
Brooke, J. SUS: a retrospective. Journal of Usability Studies 2013, 8, 29–40. [Google Scholar]
Grier, R.A.; Bangor, A.; Kortum, P.; Peres, S.C. The system usability scale: Beyond standard usability testing. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2013.
Zhang, D.; Adipat, B. Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction 2006, 18, 293–308. [Google Scholar] [CrossRef]
Tiger, L. The Pursuit of Pleasure, 1st ed.; Little, Brown: Boston, MA, 1992. [Google Scholar]
Jordan, P.W. Human factors for pleasure in product use. Applied Ergonomics 1998, 29, 25–33. [Google Scholar] [CrossRef] [PubMed]
Jordan, P.W. Designing Pleasurable Products: An Introduction to the New Human Factors; CRC Press: Boca Raton, FL, 2000. [Google Scholar]
Hassenzahl, M. The Effect of Perceived Hedonic Quality on Product Appealingness. International Journal of Human-Computer Interaction 2001, 13, 481–499. [Google Scholar] [CrossRef]
Hernández-Jorge, C.M.; Rodríguez-Hernández, A. F.; Kostiv, O.; Rivero, F.; Domínguez-Medina, R. Psychometric Properties of an Emotional Communication Questionnaire for Education and Healthcare Professionals. Education Sciences 2022, 12, 484. [Google Scholar] [CrossRef]
Artino Jr, A.R.; La Rochelle, J. S.; Dezee, K.J.; Gehlbach, H. Developing questionnaires for educational research: AMEE Guide No. 87. Medical Teacher 2014, 36, 463–474. [Google Scholar] [CrossRef]
Hardesty, D.M.; Bearden, W.O. The use of expert judges in scale development: Implications for improving face validity of measures of unobservable constructs. Journal of Business Research 2004, 57, 98–107. [Google Scholar] [CrossRef]
Natural Language API Basics. Cloud Natural Language. Available online: https://cloud.google.com/natural-language/docs/basics#interpreting_sentiment_analysis_values.
KMO and Bartlett’s Test (by IBM, 2021). Available online: https://www.ibm.com/docs/en/spss-statistics/28.0.0?topic=detection-kmo-bartletts-test (accessed on 10 May 2023).
Taber, K.S. The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education 2018, 48, 1273–1296. [Google Scholar] [CrossRef]
Guilford, J.P. Psychometric Methods, 2nd ed.; McGraw-Hill: New York, NY, 1954. [Google Scholar]
Ho, Y.; Kwon, O. Y.; Park, S.Y.; Yoon, T.Y.; Kim, Y.E. Reliability and validity test of the Korean version of Noe’s evaluation. Korean Journal of Medical Education 2017, 29, 15–26. [Google Scholar] [CrossRef]
Hu, L.T.; Bentler, P. M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 1999, 6, 1–55. [Google Scholar] [CrossRef]
Research Methods Knowledge Base (by Trochim, W.M.K., hosted by Conjointly, 2020). Available online: https://conjointly.com/kb/?_ga=2.202908566.666445287.1649411337-790067422.1649411337 (accessed on 10 May 2023).

Figure 1. The research process of this study.

Figure 3. The Emotions-as-Social-Information (EASI) model (Van Kleef [40]).

Figure 4. The “Wheel of Emotion” model proposed by Plutchik [50]. (a) The evolution process of the model. (b) The three-dimensional circumplex of the model.

Figure 5. The “Hourglass of Emotions” proposed by Cambria et al. [51].

Figure 6. Some feelings and their locations within the affect circumplex according to the factors of “pleasure” and “activation” (from Warr and Inceoglu [55]) which may be regarded to be equivalent to “valence” and “arousal,” respectively, mentioned in [52,53].

Figure 7. Multimodal interaction framework with Four-layer modular design proposed by Jonell et al. [79].

Figure 8. Framework for intelligent user interfaces (Jalil [84]).

Figure 9. The flow of research in this study.

Figure 10. The process of prototype system development in this study.

Figure 11. Hierarchical Structure of User Needs.

Figure 12. Illustration of the proposed Emotion Drift System and its experiment environment. (a) An illustration. (b) The real experiment environment.

Figure 13. Architecture of the proposed Emotion Drift System.

Figure 14. The hand movement axes principle of the Leap Motion sensor.

Figure 15. The relative spatial coordinate diagram of the Leap Motion Sensor.

Figure 17. Flowchart of semantics and sentiment analyses.

Figure 18. Mind Sensor EEG (from https://www.mindsensor.tw/). (a) The headset. (b) The sensor.

Figure 19. Emotion states and their locations within the affect circumplex according to the factors of “meditation” and “attention” (a copy of Figure 6 but with the X- and Y-axes being re-named as meditation and attention, respectively, for use in this study.).

Figure 20. User interface showing the scales of the four parameters of emotion, sensibility, relaxation, and focusing (a copy of the illustration shown in Step 4.1 of Table 8).

Figure 21. Experiment procedure of this study for the user to participate.

Figure 22. The distribution of the data pairs of semantic (listening 2) (the semantic scores for the second message shared by the previous user) as the Y-values and semantic (speaking 4) (the the semantic scores for the fourth message shared by the current user) as the X-values.

Figure 23. Results of confirmatory factor analysis (CFA) using the AMOS package. (a) Diagram of the structural model of the dimension of “system usability” generated through CFA; (b) Diagram of the structural model of the dimension of “interactive experience” generated Through CFA; (c) Diagram of the structural model of the dimension of “pleasure experience” generated Through CFA.

Table 1. High-quality connections (HQCs) and related key mechanisms.

Key Factor	Mechanism	Positive Behavior
Cognition	Personal cognition is crucial for establishing interpersonal connections. Individuals process consciousness and impressions through psychological cognition, which adjusts relationship dynamics and influences final behaviors.	attention, memory, perception, perspective-taking
Affection	Emotions arise from interactions between individuals and their environment. Emotional reactions are triggered by issues in interpersonal interactions, aiding in the understanding of emotional tendencies in relationships. The affective mechanism regulates thoughts, body, and feelings.	gratitude, empathy, compassion, concern for others
Behavior	Behavior is a key factor in interpersonal communication, reflecting actual responses or actions towards people, events, and things. These actions, driven by internal intentions, manifest externally and can significantly influence interpersonal relationships.	Respect, participation, interpersonal assistance, playful interaction

Table 2. Descriptions of Brainwave Types.

Brainwave Type	Frequency	Physiological State
δ wave	0–4 Hz	Occurring during deep sleep
θ wave.	4–8 Hz	Occurring during deep dreaming and deep meditation
α wave	8–13 Hz	Occurring when the body is relaxed and at rest.
β wave	13–30 Hz	Occurring during states of neurological alertness, focused thinking, and vigilance.

Table 3. Forms of Interaction and Applications in Affective Computing Related Cases.

Name of the work	Interaction Type	Application Area
Kismet (1999) [45]	Image Recognition, Semantic Analysis	Interpersonal Communication
eMoto (2007) [65]	Gesture Detection, Pressure Sensors	Interpersonal Communication
MobiMood (2010) [66]	Manual Input, Pressure Sensors	Interpersonal Communication
Feel Emotion (2015) [67]	Skin Conductance, Heart Rate Sensors	Assistive Psychological Diagnosis
Moon Light (2015) [68]	Skin Conductance, Heart Rate Sensors	Assistive Psychological Diagnosis
Woebot (2017) [69]	Semantic Analysis, Chatbots	Assistive Psychological Diagnosis
What is your scent today? (2018) [70]	Big Data Collection, Semantic Analysis	Tech-Art Creation

Table 4. Motivations and outcomes of related affective computing cases.

Name of the Work	Motivation of the Work	Interaction Content	Result of the Work
Kismet (1999) [45]	Exploring social behaviors between infants and caregivers and the expression of emotions through perception.	Waving and moving within the machine’s sight to provide emotional stimuli (happiness, sadness, anger, calmness, surprise, disgust, fatigue, and sleep).	Responding similarly to an infant, showing sadness without stimulation and happiness during interaction.
eMoto (2007) [65]	Starting from everyday communication scenarios and designing emotional experiences centered on the user.	Interacting by pen pressure with emotional colors based on the Emotional Circumplex Model, allowing for the free pairing of emotional colors.	Allowing users to observe and become aware of their own and others’ emotions in realtime, noting emotional shifts and changes throughout interactions.
MobiMood (2010) [66]	Using emotions as a medium for communication, aiming at enhancing interpersonal relationships.	Users can select and adjust the intensity of emotions (e.g., sadness, energy, tension, happiness, anger).	Enabling personal expression and sharing, fostering positive social interactions.
Feel Emotion (2015) [67]	Monitoring users’ physiological information to observe and adjust their emotional states.	Employing wearable devices to monitor users’ physiological information.	Sensing emotional changes in realtime to achieve proactive emotional management.
Moon Light (2015) [68]	Understanding and cultivating self-awareness and control through physiological data.	Controls the corresponding environmental light color based on the current physiological information.	Exploring how feedback from biosensor data affects interpersonal interactions.
Woebot (2017) [69]	Replacing traditional psychological counseling with digital resources.	Offering cognitive behavioral therapy through conversations and chatbot-assisted functions.	Regulating users’ emotions to counteract negative and irrational cognitive thinking and emotions.
What is your scent today? (2018) [70]	Analyzing the emotions of information from social media using classifiers.	Conducting text mining for emotions (anger, disgust, fear, happiness, sadness, surprise).	Based on analyzed emotional information, adjusting fragrance blends to reflect emotional proportions and using scents to symbolize emotions.

Table 5. Design principles for multi-sensing interfaces.

Design Principle	Description	Notes
Sensitivity	Improving the accuracy of information recognition and provide a seamless experience	Integrating human sensory input and showing natural expression characteristics
Flexibility	Avoiding similar commands and preprocess related function commands	Conducting modular system design, including expressions and operations for commanding objects
Measurement Range	Providing equipment or functions for distant control of physical or virtual objects	Using depth cameras and non-contact sensing devices

Table 6. Related cases of existing systems with multi-sensing interactive interfaces.

Name of the Work	Sensing Interface	Interaction Style	Application Domain	Affective Computing
Speech, Gesture controlled wheelchair platform (2015) [87]	Gesture Recognition, Voice Recognition, Physical Interfaces	Integrating multimodal interfaces with visual feedback for remote control using physical devices.	Intelligent life	No
SimaRobot (2016) [88]	Voice Recognition, Image Recognition	Simulating, listening to, and empathizing with users’ emotions based on personal emotional information.	Education & learning, healthcare	Yes
Gesto (2019) [89]	Voice Recognition, Gesture Recognition	Automatically executing customizable tasks.	Intelligent life	No
Emotion-based Music Composition System (2020) [90]	Physical Interfaces, Tone Recognition	Determining emotions through melody, pitch, volume, rhythm, etc.	Art creation	Yes
Industrial Robot (2020) [91]	Voice Recognition, Gesture Recognition	Using digital twin technology to achieve an integrated virtual and physical interactive experience.	Intelligent life	No
UAV multimodal interaction integration system (2022) [6]	Voice Recognition, Gesture Recognition	Controling drones and scenes within a virtual environment.	Intelligent life	No

Table 7. The dimensions and questions of the questionaire survey in this study.

Dimension	Questions
System Usability	A1. The interaction instructions help me quickly become familiar with the system.
	A2. I am able to understand and become familiar with each function of the system.
	A3. I find the system’s functionality intuitive.
	A4. The emotions output by the system seem accurate.
	A5. The process of picking up the message bottle and listening to the message is coherent.
	A6. The various functions of the system are well integrated.
	A7. I would like to use this system frequently.
Interactive Experience	B1. The system helps me care more about others.
	B2. The system helps improve my mood when I am feeling down.
	B3. I feel uncomfortable with others hearing or seeing the content I use on the system.
	B4. The system enhances my empathy.
	B5. I feel lonely while using the system.
	B6. I feel relieved not to interact face-to-face with others when using the system.
	B7. Using the system is similar to using online social networks.
Pleasure Experience	C1. I feel special when using the system.
	C2. I feel relaxed during the “rowing” interaction process.
	C3. The “sharing experiences” process makes me feel relieved.
	C4. I find the system’s interactive experience interesting.
	C5. I believe the system can enrich my life experience.
	C6. I am interested in actively using the system.
	C7. I enjoy using the system.
	C8. I easily immerse myself in the system.
	C9. Time seems to fly when I am using the system.

Table 10. Gesture Command Table.

Command	Description	Illustration
Move Forward	When both hands are clenched into fists and moved back and forth at the same rate, the game object will move forward.
Turn Left	When the right hand is clenched into a fist and swung back and forth, the game object moves left. The faster the swinging, the larger the movement angle.
Turn Right	When the left hand is clenched into a fist and swung back and forth, the game object moves right. The faster the swinging, the larger the movement angle.
Select and Click	Lightly touching the button with the index finger allows for easy navigation to other functional interfaces.
Grab Object	Grasping scene objects with a fist allows the game objects to move freely within the scene.

Table 11. Examples of emotions with corresponding numerical S and M values yielded by semantics and sentiment analyses [134].

Emotion states	Example values	Explanation
Clearly positive	score S: 0.8; magnitude M: 3.0	High positive emotion with a large amount of emotional vocabulary
Clearly nagative	score S: -0.6; magnitude M: 4.0	High negative emotion with a large amount of emotional vocabulary
Neutral	score S: 0.1; magnitude M: 0.0	Low positive emotion with relatively few emotional words
Mixed (lacking emotional expression)	score S: 0.0; magnitude M: 4.0	No obvious trend of positive or negative emotions overall. (The text expresses both positive and negative emotional vocabulary sufficiently.)

Table 12. Colored scale visualization of the sentiment-related and emotion-related data.

Original values	Value range	Scale & letter color	Color range	Meaning
Score	-1.0 ~ +1.0	red	0 ~ 255	Emotion
Magnitude	0.0 ~ +∞	transparent	0 ~ 255	Sensibility
Meditation	0 ~ 100	green	0 ~ 255	Relaxation
Attention	0 ~ 100	blue	0 ~ 255	Focusing

Table 13. Determination of classes of correlation coefficients.

Correlation class	Low Correlation	Moderate Correlation	High Correlation
Decision threshold	< 0.3	0.3 ~ 0.7	> 0.7

Table 14. Determination of significance levels of correlation coefficients.

Significance level	Significant	High significant	Very significant
Decision threshold	p < 0.05	p < 0.01	p < 0.001
Coefficient c with asterisk(s)	c*	c**	c***

Table 15. The correlation parameters of the Pearson Test between the semantic of the 2nd message left by the previous user (and listened to by the current user) and the semantic of the 4th message spoken by the current user.

Pearson Correlation Analysis	Correlation coefficient c	.266*
	Significance value p	.040
*When p < 0.05, the correlation is “significant.”

Table 16. The semantic scores of the 2nd message left by the previous user (and listened to by the current user) and the semantic of the 4th message spoken by the current user.

Interaction	Average semantic score of 50 users	Standard Deviation of semantic score of 50 users
Semantic (listening 2)	0.30	0.36
Semantic (speaking 4)	0.44	0.39

Table 17. Fifteen cases of four types of correlation of emotion data explored in this study.

Case No.	Type	Emotion dataset 1	Emotion dataset 2	Correlation coefficient	Signeficance value p	Significance level
1	1	Semantic (listening 2)	Semantic (speaking 4)	.266*	.040	significant
2	2	Semantic (listening 3)	Meditation (speaking 4)	.397**	.002	highly significant
3	2	Semantic (listening 3)	Meditation (speaking 1)	.386**	.002	highly significant
4	2	Semantic (listening 4)	Attention (speaking 1)	.285*	.027	significant
5	3	Meditation (listening 2)	Meditation (listening 3)	.394**	.002	highly significant
6	3	Meditation (listening 3)	Meditation (listening 4)	.344**	.007	highly significant
7	3	Attention (listening 1)	Attention (listening 2)	.594***	.000	very significant
8	3	Attention (listening 2)	Attention (listening 3)	.722***	.000	very significant
9	3	Attention (listening 3)	Attention (listening 4)	.667***	.000	very significant
10	4	Meditation (speaking 1)	Meditation (speaking 2)	.277*	.032	significant
11	4	Meditation (speaking 2)	Meditation (speaking 3)	.262*	.043	significant
12	4	Meditation (speaking 3)	Meditation (speaking 4)	.626***	.000	very significant
13	4	Attention (speaking 1)	Attention (speaking 2)	.477***	.000	very significant
14	4	Attention (speaking 2)	Attention (speaking 3)	.417**	.001	highly significant
15	4	Attention (speaking 3)	Attention (speaking 4)	.316*	.014	significant

Table 18. Percentages of users’ emotions in the interaction phases of message listening.

Interaction phase	Enthusiasm quadrant	Comfort quadrant	Depression quadrant	Anxiety qhadrant
Listening phase (I)	40%	20%	15%	25%
Listening phase (II)	37%	22%	10%	32%
Listening phase (III)	32%	37%	13%	18%
Listening phase (IV)	27%	37%	12%	25%

Table 19. Percentages of users’ emotions in the interaction phases of message sharing.

Interaction phase	Enthusiasm quadrant	Comfort quadrant	Depression quadrant	Anxiety qhadrant
Sharing phase (I)	20%	45%	15%	20%
Sharing phase (II)	25%	40%	17%	18%
Sharing phase (III)	25%	38%	22%	15%
Sharing phase (IV)	30%	40%	13%	17%

Table 20. Statistical summary of the basic information of the research sample.

Basic information	Category	Number of samples	Percentage
Sex	Male	22	37%
Sex	Female	38	63%
Age	18~25	46	77%
	26~35	13	22%
	36~45	1	2%
Sxperience of using dating software	Yes	47	78%
Sxperience of using dating software	No	13	22%

Table 21. Statistics of collected questionnaires according to the five-point Likert scale.

Dimension	Lebel	Min.	Max.	Mean	Standard deviation	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)
System usability	A01	3	5	4.23	0.62	33%	57%	10%	0%	0%
	A02	3	5	4.23	0.62	33%	57%	10%	0%	0%
	A03	2	5	4.28	0.86	50%	33%	12%	5%	0%
	A04	2	5	4.00	0.75	27%	48%	23%	2%	0%
	A05	3	5	4.38	0.61	45%	48%	7%	0%	0%
	A06	5	5	4.43	0.59	48%	47%	5%	0%	0%
	A07	2	5	4.00	0.78	25%	55%	15%	5%	0%
Interactive experience	B01	1	5	4.01	0.91	32%	47%	15%	5%	1%
	B02	2	5	3.9	0.83	25%	45%	25%	5%	0%
	B03	1	5	3.43	1.19	22%	30%	25%	17%	6%
	B04	2	5	4.0	0.80	30%	47%	20%	3%	0%
	B05	1	5	4.06	0.93	35%	45%	15%	2%	3%
	B06	1	5	3.97	1.13	40%	33%	15%	7%	5%
	B07	1	5	3.2	1.29	10%	48%	10%	15%	17%
Pleasure experience	C01	2	5	3.85	0.84	23%	43%	28%	5%	0%
	C02	2	5	4.31	0.81	50%	35%	12%	3%	0%
	C03	3	5	4.36	0.66	47%	43%	10%	0%	0%
	C04	2	5	4.73	0.54	77%	22%	0%	1%	0%
	C05	2	5	4.26	0.77	43%	43%	10%	3%	0%
	C06	2	5	4.23	0.76	42%	42%	15%	2%	0%
	C07	2	5	4.45	0.67	53%	40%	5%	2%	0%
	C08	1	5	4.28	0.88	48%	38%	8%	3%	2%
	C09	1	5	4.11	0.94	42%	35%	18%	3%	2%

Table 22. The KMO test measured values and the significance values of Bartlett’s test of the question answer data collected according to the three dimensions as listed in Table 21.

Dimension	Name of Measure or Test	Value
System usability	KMO measure of sampling adequacy	.786
System usability	Significance of Bartlett’s test of sphericity	.000
Interactive experience	KMO measure of sampling adequacy	.671
Interactive experience	Significance of Bartlett’s test of sphericity	.000
Pleasure experience	KMO measure of sampling adequacy	.778
Pleasure experience	Significance of Bartlett’s test of sphericity	.000

Table 23. Rotated component matrix of the 1st dimension “system usability.”.

Dimension: system usability
Question	1	2
A06	.744	.168
A03	.738	.269
A02	.733	-.004
A07	.672	.303
A04	-.004	.772
A05	.244	.764
A01	.369	.644

Table 24. Rotated component matrix of the 2nd dimension “interactive experience.”.

Dimension: interactive experience
Question	1	2
B04	.819	.178
B01	.771	-.033
B02	.725	.068
B06	.650	.190
B05	.029	.813
B03	.025	.740
B07	.257	.578

Table 25. Rotated component matrix of the 3rd dimension “pleasure experience.”.

Dimension: pleasure experience
Question	1	2	3
C04	.812	.167	.087
C05	.790	.195	.175
C09	.734	.321	-.166
C07	.173	.913	.167
C06	.398	.696	.207
C08	.565	.626	.107
C02	-.077	.325	.785
C03	.405	-.075	.759
C01	-.024	.125	.702

Table 26. Collection of the questions of the resulting latent dimensions of the three dimensions.

Dimension	Category	Latent dimension	No. of questions	Labels of questions
System usability	RA1	Functionality	4	A06, A03, A02, A07
System usability	RA2	Effectiveness	3	A04, A05, A01
Interactive experience	RB1	Empathy	4	B04, B01, B02, B06
Interactive experience	RB2	Isolation	3	B05, B03, B07
Pleasure experience	RC1	Satisfaction	3	C04, C05, C09
	RC2	Immersion	3	C07, C06, C08
	RC3	Relaxation	3	C02, C03, C01

Table 27. The questions of the three dimensions and the seven latent dimensions, as well as corresponding Cronbach’s α coefficient values computed by the SPSS.

Dimension	Category	Latent dimension	Cronbach’s α Coefficient of the latent dimension	Cronbach’s α Coefficient of the dimension
System usability	RA1	Functionality	.735	.786
System usability	RA2	Effectiveness	.604	.786
Interactive experience	RB1	Empathy	.711	.671
Interactive experience	RB2	Isolation	.577	.671
Pleasure experience	RC1	Satisfaction	.694	.778
	RC2	Immersion	.694
	RC3	Relaxation	.638

Table 28. Fitness indexes of the structural models of the three dimensions of questions generated by the CFA.

Dimension	df	χ²	χ²/df	cfi	RMSEA	RMSEA (90% CI)
Dimension	df	χ²	χ²/df	cfi	RMSEA	LO	HI
System usability	13	6.971	0.536	1.000	0.000	0.000	0.054
Interactive experience	13	11.170	0.859	1.000	0.000	0.000	0.113
Pleasure experience	23	31.726	1.379	0.951	0.080	0.000	0.143
^a Meanings of symbols—df: degree of freedom; gfi: goodness-of-fit index; agfi: average gfi; cfi: comparative fit index; RMSEA: root mean square error of approximation; CI: confidence interval; LO: low; HI: high.

Table 29. The construct validity values of the latent dimensions of the three dimensions of “system usability,” “interactive experience,” and “pleasure experience” generated through CFA.

Dimension	Category	Group of Related Questions	Construct Validity Value
System usability	RA1	A06, A03, A02, A07	0.742
System usability	RA2	A04, A05, A01	0.650
Interactive experience	RB1	B04, B01, B02, B06	0.750
Interactive experience	RB2	B05, B03, B07	0.573
Pleasure experience	RC1	C04, C05, C09	0.771
	RC2	C07, C06, C08	0.830
	RC3	C02, C03, C01	0.677

Table 30. Analysis of the responses to the questions of the latent dimension of “functionality.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
A06	The various functions of the system are well integrated.	5	5	4.43	0.59	48%	47%	5%	0%	0%	95%
A03	I find the system’s functionality intuitive.	2	5	4.28	0.86	50%	33%	12%	5%	0%	83%
A02	I am able to understand and become familiar with each function of the system.	3	5	4.23	0.62	33%	57%	10%	0%	0%	90%
A07	I would like to use this system frequently.	2	5	4.00	0.78	25%	55%	15%	5%	0%	80%

Table 31. Analysis of the responses to the questions of the latent dimension of “effectiveness.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
A04	The emotions output by the system seem accurate.	2	5	4.00	0.75	27%	48%	23%	2%	0%	75%
A05	The process of picking up the message bottle and listening to the message is coherent.	3	5	4.38	0.61	45%	48%	7%	0%	0%	93%
A01	The interaction instructions help me quickly become familiar with the system.	3	5	4.23	0.62	33%	57%	10%	0%	0%	90%

Table 32. Analysis of the responses to the questions of the latent dimension of “empathy.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
B04	The system enhances my empathy.	2	5	4.0	0.80	30%	47%	20%	3%	0%	77%
B01	The system helps me care more about others.	1	5	4.01	0.91	32%	47%	15%	5%	1%	79%
B02	The system helps improve my mood when I am feeling down.	2	5	3.9	0.83	25%	45%	25%	5%	0%	70%
B06	I feel relieved not to interact face-to-face with others when using the system.	1	5	3.97	1.13	40%	33%	15%	7%	5%	73%

Table 33. Analysis of the responses to the questions of the latent dimension of “isolation.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
B05	I feel lonely while using the system.	1	5	4.06	0.93	35%	45%	15%	2%	3%	80%
B03	I feel uncomfortable with others hearing or seeing the content I use on the system.	1	5	3.43	1.19	22%	30%	25%	17%	6%	52%
B07	Using the system is similar to using online social networks.	1	5	3.2	1.29	10%	48%	10%	15%	17%	58%

Table 34. Analysis of the responses to the questions of the latent dimension of “satisfaction.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
C04	I find the system’s interactive experience interesting.	2	5	4.73	0.54	77%	22%	0%	1%	0%	99%
C05	I believe the system can enrich my life experience.	2	5	4.26	0.77	43%	43%	10%	3%	0%	86%
C09	Time seems to fly when I am using the system.	1	5	4.11	0.94	42%	35%	18%	3%	2%	77%

Table 35. Analysis of the responses to the questions of the latent dimension of “immersion.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
C07	I enjoy using the system.	2	5	4.23	0.76	42%	42%	15%	2%	0%	84%
C06	I am interested in actively using the system.	2	5	4.45	0.67	53%	40%	5%	2%	0%	93%
C08	I easily immerse myself in the system.	1	5	4.28	0.88	48%	38%	8%	3%	2%	86%

Table 36. Analysis of the responses to the questions of the latent dimension of “relaxation.”.

Label	Question	Min.	Max.	Mean	S. D.	Strongly agree (5)	Agree (4)	Average (3)	Disagree (2)	Strongly disagree (1)	Agree or above
C02	I feel relaxed during the “rowing” interaction process.	2	5	4.31	0.81	50%	35%	12%	3%	0%	85%
C03	The “sharing experiences” process makes me feel relieved.	3	5	4.36	0.66	47%	43%	10%	0%	0%	90%
C01	I feel special when using the system.	2	5	3.85	0.84	23%	43%	28%	5%	0%	66%

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.

MDPI Initiatives

Important Links

Choose an area of interest and we will send you notifications of new preprints at your preferred frequency.

Disclaimer

Affective Computing Using an Emotion-Perception System for Interactive Experiences with Multi-Sensing Interfaces

Abstract

1. Introduction

1.1. Background

1.2. Research Motivation

1.3. Research Problems and Purposes

1.4. Research Scope and Limitations

1.5. Research Process

2. Literature Review

2.1. Interpersonal Interaction

2.1.1. Interpersonal Interaction Needs and Mechanisms

2.1.2. Perceived Interest in Interpersonal Interaction

2.1.3. A Summary

2.2. Affective Computing

2.2.1. Development of Affective Computing

2.2.2. Related Cases of Affective Computing

2.2.3. A Summary

2.3. Multi-Sensing Interactive Interfaces

2.3.1. Needs of Multi-Sensing Interactive Interfaces

2.3.2. Related Cases of Multi-Sensing Interactive Interfaces

2.3.3. A Summary

2.4. Concluding Remarks of the Literature Review

3. Methodology

3.1. Selection of Research Methods

3.2. The Affinity Diagram Method

3.2.1. Idea of Using the Affinity Diagram Method

3.2.2. Application in This Study and Brainstorming Result

3.3. The Prototype Development Method

3.4. Indirect Observation Method

3.5. Questionnaire Survey Method

3.5.1. Ideas and Applications of the Method

3.5.2. Purpose of Survey and Question Design

4. System Design

4.1. The Design Concept of the Proposed System

4.2. Design of the Multi-Sensing Interaction Process for the Proposed System

4.2.1. Idea of the Design

4.2.2. Detailed Description of the Interaction Process

4.2.3. Questions to Answer as Messages for User Interactions

4.3. System Architecture and Employed Technologies

4.3.1. Overview of the Architecture

4.3.2. Leap Motion Hand Gesture Sensor

4.3.3. Google Speech API

4.3.4. Mind Sensor EEG

4.3.5. Visual Displays of System Data and Verbal Descriptions of Users’ Emotion States

5. System Experience Experiment and Experimental Data Analysis

5.1. System Exhibition and Experiment Processes

5.2. Indirect Observation and Analysis of Collected Data

5.2.1. Ideas of Applying the Indirect Observation Method

5.2.2. Statistical Analysis of Correlations of Semantic and Brainwave Data

5.2.3. Concluding Remarks of Statistical Analysis

5.2.4. Observations of Emotion Changes in Listening and Sharing

5.2.5. Concluding Remarks about Emotion Changes while Using the System

5.3. Questionnaire Survey and Statisitcal Analysis of Answer Data

5.3.1. Statistics of Participants in the Questionnaire Survey

5.3.2. Design of Questionnaire

5.3.3. Analysis of Reliability and Validity of Collected Questionnaire Data

5.3.4. Detailed Analyses of Answer Data of the Dimension of System Usability

5.3.5. Detailed Analyses of Answer Data of the Dimension of Interactive Experience

5.3.6. Detailed Analyses of Answer Data of the Dimension of Pleasure Experience

5.3.7. Concluding Remarks about the Analysis of the Questionnaire Survey Results

6. Conclusions and Suggestions

6.1. Conclusions

6.2. Suggestions for Future Research

Author Contributions

Funding

Informed Consent Statement

Data Availability Statement

Conflicts of Interest

References

MDPI Initiatives

Important Links

Subscribe