Affective Computing Using an Emotion-Perception System for Interactive Experiences with Multi-Sensing Interfaces

1. Introduction

1.1. Background

The COVID-19 pandemic has led to widespread mental health issues, including depression, anxiety, and loneliness. Dubé et al. [1] found a strong link between pandemic-induced social problems and negative emotional behaviors such as depression. As the pandemic eased, social media became a common way to seek comfort and connection, offering low-cost, flexible, and unrestricted interaction, thus significantly enhancing interpersonal connections [2]. The World Health Organization (WHO) has designated October 10th each year as “World Mental Health Day” to emphasize the importance of mental health in shaping our thoughts, feelings, and behaviors. The day encourages people globally to foster positive, relaxed, and constructive emotions [3].

As AI evolves, high-efficiency computing alone no longer meets user needs. Users increasingly seek machines that can understand emotions, driving demand for emotional sensing technology. Affective computing, an interdisciplinary field involving physiology, sociology, cognitive science, mathematics, psychology, linguistics, and computer science, holds significant research value. It uses various electronic materials to sense and collect emotional data from voices, facial expressions, and behaviors, creating systems for emotion recognition [4].

Collins [5] proposes in his interaction ritual chains theory that rituals are normative behaviors that foster mutual attention and emotional resonance through rhythmic interactions. This theory demonstrates that both the content and context of interactions influence the overall experience. When combined with sociological theories, it reveals that interactions involve emotions, which subsequently generate a series of rituals [6].

Users’ cognitive psychology influences their interaction behaviors, with emotional factors playing a significant role. Empathy and machines’ understanding of human emotions are becoming integral to modern life [7]. Affective computing systems improve interaction experiences by understanding users’ psychological states, highlighting the need for a human-centered approach. MIT Media Lab research shows that affective computing enhances self-awareness, communication, and stress reduction [8]. As an emerging technology, affective computing offers real-time status detection and personalized services. It is aimed in this study to integrate affective computing into interpersonal interactions, providing timely assistance and decision-making to enhance the quality of people’s life.

Machine sensing methods fall into three categories: (1) Facial image-based sensing, such as eye trackers, which distinguish facial features from noise; (2) Voice and brainwave recognition, enhancing the reliability of physiological information beyond visual data; and (3) Object detection sensors, like infrared sensors, which use multi-sensing approaches to optimize human-machine interaction and feedback [9].

Affective computing analyzes human social behaviors and emotional information to provide appropriate feedback through human-machine interaction [10]. People use gestures and language for daily communication, and combining these modes offers high usability. Gestures can convey commands that are difficult to express verbally, simplifying and enhancing remote human-machine collaboration [11].

In summary, the aim of this study is to integrate affective computing with multi-sensing interactive interface technology to propose an emotion-aware interactive system characterized by pleasure, interactivity, and usability. The system will be tested in interpersonal interaction scenarios within smart living contexts to assess whether the criteria for usability, interactivity, and pleasantness are met.

1.2. Research Motivation

The COVID-19 pandemic has restricted global activities, boosting demand for online digitalization and shifting most interactions to virtual forms [12]. This has accelerated the development of physical digitization and contactless methods, overcoming time and space limitations. As the post-pandemic situation stabilizes, new interaction models, both proximate and remote, have emerged. Mixed-reality multi-sensing interfaces are subtly transforming daily life experiences.

Advances in artificial intelligence have strengthened the link between humans and smart technology, highlighting the growing importance of affective computing. There is increasing focusing on ensuring interactions to be friendly and enjoyable, with human-machine collaboration aimed at delivering warmer, higher-quality services.

According to Coherent Market Insights [13], the affective computing market, valued at $36.32 billion in 2021, is projected to reach $416.9 billion by 2030, with a compound annual growth rate of 31.5% from 2022 to 2030. The shift from rational to emotional user needs is driving leading technology companies to focus on the use of computational data for product design, a trend expected to continue.

Through interactions within specific contexts, novel situations and emotions can be evoked, creating fresh meanings. Human-machine interaction mechanisms are aimed to be established in this study using theories such as Interaction Ritual Chains and Situational Awareness. Leveraging the rise of affective computing technology, emotions are intended to be identified and recorded through interactions. Machine recognition of human emotions will be explored to enable diverse feedback tailored to different outcomes, enhancing personalized experiences and promoting greater self-awareness. It is noted that current designs of voice interaction systems utilizing multi-sensing technology are still limited.

Furthermore, it has been noted that emotional exchanges in Computer-Mediated Communication (CMC) environments are more frequent and clearer than in face-to-face (F2F) settings, with positive emotions being effectively amplified. Social media also enhances human interactions by offering low cost, flexible norms, and removing spatial and temporal constraints.

In summary, this study is aimed at developing a design system to enhance positive emotions in interpersonal interactions through the described mechanisms and to further investigate emotion transmission in communication. he integration of voice recognition into interaction patterns will also be explored to propose a novel system combining multi-sensing interactive interfaces with affective computing. The system will be applied to various smart living scenarios.

1.3. Research Problems and Purposes

In this study, the literature analysis has revealed that few interactive systems combine multi-sensing interaction with affective computing technologies. The issue of translating and reproducing users’ complex emotions by collecting emotional data from their interactions is to be explored. The goal is to help users better understand their own emotions through the system, making the experience more intuitive and relevant.

The questions studied in this research are as follows.

(1)

How can the integration of multi-sensing interaction and affective computing technologies enhance interpersonal relationships?

(2)

Does constructing an interactive system based on multi-sensing interaction and affective computing technologies strengthen positive interpersonal communication experiences and stimulate positive emotion transmission?

(3)

Can the features of multi-sensing interaction and affective computing technologies improve the pleasantness of the interaction experience?

In this study, it is aimed to explore how to create pleasant interactive experiences for users through theoretical exploration and case analysis. An interactive system based on multi-sensing interfaces and affective computing technology is to be developed to enhance interpersonal interactions. Additionally, interaction mechanisms are to be investigated to promote positive emotional effects and examine emotion transmission in interpersonal interactions.

The research objectives are as follows.

(1)

To analyze cases of multi-sensing interaction and affective computing, and summarize design principles for the proposed interactive system.

(2)

To develop an emotion-aware interactive system that enhances positive interpersonal relationships.

(3)

To evaluate whether the system effectively makes users feel pleasant and relaxed.

1.4. Research Scope and Limitations

In this study, the investigation will focus on interactive experiences using multi-sensing interfaces and affective computing. The scope and limitations are as follows:

(1)

Voice recognition and Leap Motion gesture sensing are selected as the primary multi-sensing interfaces.

(2)

Explore emotion recognition in interactive experiences will be explored, but the development of semantic analysis technologies will not be addressed.

(3)

The designed multi-sensing system will be tested with users who have intact limbs, good auditory and visual abilities, and understand typical interpersonal interactions, specifically targeting active users aged 18 to 34.

1.5. Research Process

The research process, illustrated in Figure 1, is divided into five stages, with each stage’s details described below.

Research process of this study -

(1)

Stage 1 - Research concept development:

(1.1) Understanding the research background and motivation.

(1.2) Defining the research purpose.

(2)

Stage 2 - Literature review and case analysis:

Conducting analysis in three main areas: interpersonal interaction, affective computing, and multi-sensing interactive interfaces, as shown in Figure 2.

(3)

Stage 3 – System planning and construction:

(3.1) Deriving design principles for multi-sensing interaction, including summarizing properties and proposing man-machine interaction architecture.

(3.2) Developing a prototype system using affinity diagrams.

(3.3) Constructing the final multi-sensing interactive system.

(4)

Stage 4 – Exhibition and evaluation:

(4.1) Conducting public exhibition and experiments.

(4.2) Evaluating the proposed system by the methods of affinity diagrams and questionnaire survey.

(5)

Stage 5 – Ending:

Drawing conclusions and making suggestions for future research.

Figure 2. Main areas for the literature review and research in this study.

Figure 2. Main areas for the literature review and research in this study.

2. Literature Review

As previously mentioned, the focus of this study is on interpersonal interaction, affective computing, and multi-sensing interactive interfaces, as shown in Figure 2. The literature review for each area begins with definitions and key issues (needs, mechanisms, developments) and is followed by a summary of findings that are used as a reference for the system design.

2.1. Interpersonal Interaction

A literature review is conducted focusing on emotional communication within interpersonal interactions, emphasizing interactive patterns and emotional perceptions between individuals. Research related to interpersonal interactions, both past and present, is examined to explore their content and characteristics. From this analysis, relevant design principles for the proposed system are derived.

2.1.1. Interpersonal Interaction Needs and Mechanisms

Interpersonal interaction, or interpersonal communication, involves the exchange of language, thoughts, and feelings between individuals. According to DeVito [14], interpersonal relationships serve four key functions: alleviating loneliness, securing stimulation, fulfilling needs, and providing contact for self-knowledge. These functions explain the motivation behind developing interpersonal relationships.

Interpersonal interaction is divided into verbal communication (e.g., face-to-face conversations, phone calls) and non-verbal communication (e.g., body language, facial expressions). Effective interaction is characterized by clear communication, active listening, and mutual understanding and respect. In daily life, communication generates symbols, leading to meaningful behaviors and reflecting new concepts and trends [15]. The processes of encoding (creating messages) and decoding (interpreting messages) are involved in communication [16]. According to Norman [17], the reliance on technology now spans information services such as communication, news, entertainment, education, and social interaction. Methods of communication, including symbols, sounds, text, and other perceivable signals, are used as valuable symbols in interpersonal interaction.

Today, multimedia communication has transformed interactions, connecting individuals across various locations and representing a new trend. Research shows that individuals aged 18 to 29 are the most active on social media, with approximately 84% of adults in this age group reporting use of at least one platform in 2021 [2,18]. Digital technologies have introduced various forms of interpersonal interaction, including real-time messaging apps, chatbots, and two-way audio-visual transmission.

Derks et al. [19] introduced computer-mediated communication (CMC), which uses computer networks to exchange digital messages and data. Their research indicates that emotional communication in CMC environments is more frequent and clearer than in face-to-face settings, potentially amplifying positive emotions [20]. Li et al. [21] found that social media, as a form of interpersonal interaction, addresses user needs such as finding common interests, anonymous participation, flexible norms, and overcoming spatial and temporal constraints, thus enhancing interactions.

In interpersonal communication, emotional contagion, or the “chameleon effect,” involves unconsciously imitating others’ emotions and behaviors. Research shows that individuals with higher empathy are more likely to exhibit this effect [22]. Emotional contagion can strengthen relationships and foster empathy, which aids in achieving shared goals [23].

Studies suggest that synchronized actions form a strong impression of interaction partners’ behaviors. Positive emotional contagion not only improves interactive task effectiveness but also enhances perspective-taking and empathy. Thus, a harmonious atmosphere, positive emotions, and attractiveness are crucial for future research [24]. High-quality connections (HQCs), as detailed in Table 1, are short-term, bidirectional interactions that foster engagement, openness, and positivity, with cognitive, emotional, and behavioral mechanisms being key [25].

In summary, understanding the connection between user motivations, emotions, and cognition is crucial for enhancing relationships and influencing continued system use. Cognitive interest criteria in interpersonal interactions will be discussed, and the derived design principles will be summarized in later sections.

2.1.2. Perceived Interest in Interpersonal Interaction

Jiarui et al. [26] highlighted that traditional research on interpersonal interactions often focused on single dimensions of human-to-human interaction. The integration of human, cognitive, and system interaction effects was attempted, and Davis’s technology acceptance model (TAM) was applied to explore perceived usefulness (PU) and perceived ease of use (PEOU) [27]. It was revealed that social network participation is driven by functional, social, psychological, and hedonic needs, demonstrating that emotions significantly impact interpersonal interaction intentions. Further investigation into emotional perception through the cognitive aspect of perceived interest will be conducted in this study.

In recent years, emotions like enjoyment, pleasure, and interest have been recognized as crucial for users’ intentions to continue using technology. Perceived enjoyment (PE) has been shown to significantly affect users’ attitudes and behavioral intentions towards technology, influencing actual usage behavior [28]. Venkatesh et al. [29] categorized enjoyment as intrinsic motivation and perceived usefulness (PU) as extrinsic, noting that enjoyment enhances ease-of-use and, in turn, perceived usefulness. Mun and Hwang [30] found that in enjoyable contexts, perceived enjoyment (PE) is a stronger predictor of usefulness (PU) than ease-of-use (PEOU).

Chung and Tan [31] identified key factors affecting users’ perceived enjoyment, with intrinsic motivation being central. They highlighted that content quality, ease of use, feedback diversity, usefulness, and pleasure are crucial for intrinsic motivation and should guide design processes. Brown et al. [32] introduced the situated cognition theory, which posits that cognition, thinking, and learning are context-dependent. This theory emphasizes the role of specific situations, spaces, and cultural contexts in shaping human knowledge and stresses the importance of designing interactions that align with socio-cultural contexts for effective cognitive and sensory management [33].

Furthermore, Schrepp et al. [34] compiled multimedia design guidelines, including: (1) minimizing visual clutter; (2) facilitating cognitive processes for information selection, organization, and integration; (3) enabling interaction with the environment and objects; (4) presenting multimodal information (text, images, sound); and (5) using animations appropriately. These guidelines offer preliminary directions for designing the interactive system in this research.

Collins [35] developed the interaction ritual chain theory by combining Goffman’s symbolic interaction theory with Durkheim’s social facts theory. This theory emphasizes “situational” perception and emotional aspects often neglected in social network studies. It suggests that rituals, through rhythmic interactions, focus users’ attention and emotional resonance, impacting the experience’s value. Collins [5] further highlighted that shared goals and emotions in interactions extend into daily life.

Bellocchi et al. [36] found that behavior, emotions, and cognition are interconnected. Interaction standards emerge from users’ collaboration and emotional resonance, with positive emotions indicating enjoyment and sustaining interactions. The interaction ritual chain theory’s uniqueness lies in its emphasis on users’ experiences and shared emotions, which foster group identification and enhance emotional energy, promoting positive emotions [37].

Patricia [38] challenged traditional views by noting that IoT technology has expanded digital social interactions, including communication and entertainment. Even without physical contact, emotional energy persists in virtual spaces like chatrooms, where users express emotions through images or text. This evolution of interaction ritual chain theory underscores the significance of feeling “understood” in strengthening relationships and improving interactions on social networks. Both situated cognition theory and interaction ritual chain theory enhance interpersonal interactions in virtual contexts.

Clark [39] noted that language uses reflect personal and social dimensions and emphasized that languages in interpersonal contexts involve collaborative narrative between speakers and listeners, with “understanding others” being central. Given the ambiguous nature of information in these contexts, understanding relies on various cues to extract key information.

Van Kleef [40] introduced the Emotions-as-Social-Information (EASI) model as illustrated in Figure 3, which posits that interpersonal emotional influences occur through reasoning processes or emotional stimuli. This model differentiates emotions based on the observer’s mood, indicating that discrete emotions (joy, anger, disgust, fear, sadness) serve a purposeful function, revealing the observer’s intentions or motivations in response to events.

Van Kleef et al. [41] validated the importance of emotions in situational contexts, emphasizing that decision-making in interpersonal interactions is often driven by emotions. Their research shows that emotions highlight event relevance and guide subsequent behaviors, including empathy and emotional contagion. Social interactions are identified as the primary contexts for expressing and sharing emotions, where individuals communicate their emotions and interpret those of others.

Wright and McCarthy [42] noted that most users rely on portable devices for information and entertainment, with instant messaging being a common mode of social interaction. Social apps like email, LinkedIn, Facebook, and Twitter use notification badges to prompt urgent responses, which can create anxiety and reduce interaction quality. The study suggests that slower messaging methods could alleviate stress and enhance communication by incorporating diverse perspectives, such as contextual, narrative, and autobiographical.

Hallnäs and Redström [43] argued that interaction design has evolved from creating efficient tools to developing experiences and smart technologies integrated into daily life. They introduced “slow technology,” which emphasizes moments for reflection and mental rest. Slow technology encourages thoughtful engagement and suggests that slower interactions, such as text-based communication, are crucial for reflective language processing and providing space for contemplation.

Therefore, in this study, an attempt is made to integrate the indicators of “Functionality,” “Usability,” and “Enjoyability” mentioned in the literature with concepts from theories such as situated cognition, interaction ritual chain, and emotions-as-social-information model. The system design principles derived are as follows:

(1)

Effectiveness: Providing users with systems and functions that prioritize immediacy and convenience.

(2)

Enjoyability: Considering interface diversity and emotional resonance to enhance user focusing and immersion.

(3)

Simplicity: Offering a straightforward, intuitive, and natural user interface.

(4)

Interactive Context: Recognizing the significance of multimedia virtual spaces in daily life and leveraging social interaction behaviors to stimulate users’ emotional awareness and behaviors.

2.1.3. A Summary

In this study, the broad definition of interpersonal interaction is explored. According to the literature surveyed, interactions primarily involve fixed, repetitive actions aimed at achieving expected goals. Scholars have identified that social network interactions fulfill functional, social, psychological, and hedonic needs. Design considerations must incorporate intrinsic user motivations. The situated cognition theory and interaction ritual chain theory underscore that in digital multimedia social spaces, users engage in real-time communication through voice and text, fostering emotional resonance. The Technology Acceptance Model suggests a positive relationship between usability and enjoyment, which will be detailed further in Section 4.

To design effective virtual social network interactions, incorporating these principles is essential. This approach aims to enhance users’ interaction experiences and foster continued engagement intentions. The study’s primary objective is to develop a system that enriches interpersonal interactions with novelty and enjoyment.

Key findings from this section include the following points.

(1)

Emotional resonance and empathy: Interpersonal interactions evoke emotional resonance and empathy, fostering positive feelings through active communication.

(2)

Intrinsic motivation in interaction contexts: Exploring intrinsic motivations within online contexts facilitates emotional exchanges and enjoyable flow experiences.

(3)

Real-time interaction in digital multimedia spaces: These spaces enable interactions across formats, times, and locations, promoting continuous participation.

(4)

Integration of cognitive and emotional aspects: Combining these aspects enhances system usability, addressing needs such as loneliness alleviation and self-understanding.

(5)

Relationship between usability, enjoyment, and interaction quality: The system’s usability and enjoyment directly impact the quality of interpersonal interactions.

2.2. Affective Computing

2.2.1. Development of Affective Computing

The term affective computing was introduced by Picard [44] in 1995. This field combines psychology, cognitive science, and computer science to address emotional needs by detecting physiological data, also known as artificial emotional intelligence.

Norman [17] observed that while people use bodily movements to interact with machines, machines use sounds, tactile feedback, and displays to communicate. Advances in artificial intelligence have enabled machines to perform tasks once exclusive to humans. As machine learning and communication technologies advance, emotion-sensing technologies are increasingly used to interpret psychological states, leading to more human-like affective computing systems.

Picard [44] proposed a four-level framework for affective computing, which aims to develop systems that can recognize, interpret, process, and simulate human emotions:

(1)

Recognizing Emotion: Analyzing physiological signals, voice, facial expressions, and text to determine users’ emotional states. This is the most widely applied level.

(2)

Expressing Emotion: Enabling computers to communicate with users by expressing emotions. This level often involves simulation based on predefined settings, commonly seen in chatbots and virtual characters.

(3)

Having Emotion: Allowing computers to possess emotions similar to humans, leading to emotional decision-making and actions. This level is still developing and requires careful management.

(4)

Showing Emotional Intelligence: Computers can self-regulate, moderate, and manage their emotions, effectively functioning as “friendly artificial intelligence.”

Currently, most research in affective computing focuses on the first level, which is the foundational aspect of the field. The second level, involving the expression of emotions by computers, requires interfaces that simulate emotional expressions for effective user communication. Scholars recommend designing systems to recognize and express emotions akin to human interactions.

Early work by Breazeal and Scassellati [45] led to the development of the social robot Kismet, which mimicked human emotional communication through facial expressions and sounds. Kismet combined visual perception, attention, motivation, behavior, and interaction systems to create meaningful and intentional interactions, resembling human caregiving for infants.

Emotion is crucial in human cognitive processes and enhances both performance and decision-making capabilities in systems. Affective computing improves upon traditional interactions by making human-computer interfaces more accurate and engaging. According to Shukla et al. [46], emotion recognition is a trend that will become increasingly significant in the near future, offering personalized user experiences on social media. Despite this, the emotional aspects of social media behavior are often overlooked [47]. By enabling computers to recognize, categorize, and respond to human emotions, affective computing reduces frustration and aids in understanding both personal and others’ emotions [48].

Sloman [49] distinguished between primary emotions (e.g., fear, joy), which are instinctive and detectable through facial or bodily sensors, and secondary emotions, which are more complex and influenced by primary emotions. Secondary emotions, detected through semantic analysis, are often overlooked but are crucial in daily life. Understanding these emotions is essential, as detecting emotions solely through physiological appearance is insufficient.

Plutchik’s “Wheel of Emotions” model [50] pairs eight primary emotions in opposites: joy vs. sadness, anger vs. fear, trust vs. distrust, and surprise vs. anticipation. Secondary emotions arise from these primary ones. The model, illustrated in Figure 4a,b, represents emotions as a continuous process from stimulus events through cognition, sensation, impulsive action, and effects, providing a framework for understanding the relationships among various emotional states.

In subsequent research, Cambria et al. [51] expanded on Plutchik’s “Wheel of Emotions” model [50] by introducing the “Hourglass of Emotions” model (see Figure 5). This model arranges emotional states around four primary emotions, forming an hourglass shape that represents emotional intensity across four dimensions. The vertical axis shows the intensity of different emotional dimensions, while the horizontal axis reflects the intensity of emotions triggered by various stimuli. The Hourglass of Emotions model can assess human-computer interactions by measuring user pleasure (pleasantness), attention, sensitivity, and adaptability. It detects the current intensity of a user’s emotions through these dimensions.

Plutchik’s “Wheel of Emotions” model allows for observing primary emotions and their opposites while exploring various intensities and correlations between emotions. Feldman [52] notes that Russell’s early work [53] identified emotions based on “valence” and “arousal,” represented on horizontal and vertical axes, respectively, of an affect circumplex. The arousal dimension measures the level of emotional stimulation, ranging from calm and relaxed to excited and agitated. When combined with the valence dimension, which ranges from highly positive to highly negative, these dimensions effectively capture a user’s emotional state [53,56]. To understand emotion relationships, studies [50,54,55] have used similar scaling methods to position emotions on the circumplex, as shown in Figure 6. This approach involves selecting and explaining emotional states from a broad range of terms to aid in emotion validation and analysis, and is employed in this study.

Researchers often categorize emotional stimuli into four types based on valence and arousal dimensions as follows. (1) High Valence, High Arousal (HVHA): Positive and highly stimulating emotions. (2) High Valence, Low Arousal (HVLA): Positive but less stimulating emotions. (3) Low Valence, High Arousal (LVHA): Negative but highly stimulating emotions. (4) Low Valence, Low Arousal (LVLA): Negative and less stimulating emotions. Experiments expose participants to stimuli like sadness (negative, low arousal), calmness (positive, low arousal), anxiety (negative, high arousal), and happiness (positive, high arousal), plus a neutral condition, to observe and verify emotional state changes through physiological sensors [57,58,59,60].

Wang and Chen [61] created a system that visualizes and senses emotional states using physiological data. An EEG device monitors emotional states during drawing activities, translating EEG signals into various emotions (focus, relaxation, calmness, anxiety) and providing feedback where EEG signals include δ, θ, α, and β waves [62,63,64], as described in Table 2.

2.2.2. Related Cases of Affective Computing

The COVID-19 pandemic has accelerated the integration of affective computing into interpersonal interactions, making it a significant technological trend. This has resulted in more immediate, diverse, and innovative digital user experiences. This study analyzed several cases of affective computing in interpersonal interactions, comparing their content, forms, and applications, as detailed in Table 3. Additionally, the study examined the relationships between research motivations, content, and results in these cases, as summarized in Table 4.

In summary, emotions subtly shape daily life and relationships, with individuals constantly evaluating their emotional states through various systems. Affective computing leverages these emotions to assist with decision-making, emotional regulation, and stimulating emotional resonance. Boehner et al. [71] observed that emotions are subjective and arise from interactions with others. They categorized the effects of different media on emotional experiences into several areas: biological sensing, information generation, social and cultural factors, interaction behavior, and design and evaluation. They also offered related findings and recommendations as described in the following.

(1)

Users construct emotional cognitive experiences based on information about interpersonal interactions.

(2)

Emphasis may be placed on how users experience and understand emotions through system interactions.

(3)

Emotional data should be situated within emotional dimensions to allow for free and rich emotional expression.

(4)

The aim is to develop systems that provide users with meaningful emotional stimulation.

(5)

Evaluation focuses on detecting and interpreting how users’ emotions change over time and with different stimuli.

According to Gartner [72], emotional artificial intelligence is a new addition to the emerging technology maturity curve, expected to peak within five to ten years. This technology is set to profoundly impact society. As an innovative human-computer interaction approach, affective computing provides design and evaluation strategies for co-creation and emotional experience, helping machines better understand human emotions and allowing individuals to better grasp their own and others’ emotions, thus enhancing decision-making and emotional expression.

2.2.3. A Summary

The key points from this literature review are as follows.

(1)

Applications in communication: Affective computing is widely applied in interpersonal communication, with remote tools enhancing the detection of users’ emotional states.

(2)

Empathy and emotional resonance: It enables computers to understand and express emotions, fostering empathy, relaxation, and focusing in users.

(3)

Case study insights: Case studies commonly aim to improve negative emotions and enhance empathic communication, focusing on a better quality of life and human-centered experiences through techniques like semantic analysis.

(4)

Encouraging interaction: System designs should promote user interaction and emotional awareness, offering cues that stimulate reflection without overly defining emotions.

It is observed that affective computing cannot rely on a single sensor to fully collect and interpret emotional data for users. Instead, integrating multiple sensor interfaces is necessary for comprehensive emotional data collection and detection. The next step will be to explore interactive interfaces with multiple types of sensors.

2.3. Multi-Sensing Interactive Interfaces

2.3.1. Needs of Multi-Sensing Interactive Interfaces

With advanced sensors now common in electronic devices, a wealth of emotional information is available in social network communications, including audio and video. Poria et al. [73] found that interactions have shifted from single to multi-modal approaches with the rise of the Internet. While most research focuses on audiovisual sensing, there is limited work combining semantic analysis with audiovisual data. Poria et al. [74] highlight that natural human communication is inherently multi-modal, involving text, sound, and visuals. Effective research should integrate these modalities to capture and convey semantic and emotional information seamlessly.

Fang et al. [75] identified multimodal sensing and integration with human sensory inputs as a promising research direction. Single-modal sensors can be prone to errors and noise, making multi-sensory feedback a valuable complement. They noted that finger and wrist angle detection in hand movement technology is increasingly popular. In smart living applications with many interconnected devices, this integration enhances user convenience and flexibility. The study emphasizes the importance of clear gesture recognition, recommending that similar gestures be differentiated to prevent errors and that meaningless gestures be redefined for improved user experience [76].

Due and Licoppe [77] highlight that gestures act as indicative and symbolic signs, with their meanings shaped by context. Their research suggests that multimodal sensing, informed by Gestalt psychology, enhances interaction by integrating users’ past experiences with the current gesture shape and position. This approach fosters more intuitive interactions, even without verbal communication.

Ferscha et al. [78] support that gestures offer intuitive command interactions. They emphasize the need for standardized and flexible gesture commands, focusing on directional position sensing to enhance naturalness and immersion. Designing paired commands (e.g., start/pause, left/right) and handling complex hand gestures improves interaction efficiency and integration across different sensors.

To express thoughts and emotions, people rely on sharing, communication, and message transmission, necessitating accurate sensor data capture. Jonell et al. [79] introduced a multi-sensing interaction framework that integrates voice, motion, and other sensory inputs (see Figure 7). This modular design approach involves breaking the system into units that process sensor data sequentially. The framework includes four layers: sensor API module, sensor data preprocessing module, information processing module, and end user. Voice data, captured by a microphone sensor, is processed by the preprocessor module for behavior detection or visual processing, and then delivered to the end user. The results show that a modular system enhances flexibility and intuitiveness in multi-sensing interactions.

Ueng and Chen [80] proposed a multimodal system using computer vision for environmental interaction. They observed that users typically interact with screens from 80–90 centimeters away and perform hand gestures within 50 centimeters of the camera. Leap Motion, known for its high gesture detection capabilities, is frequently used in these contexts and often combined with voice sensors for a seamless, user-friendly interface in smart home applications.

Yasen [81] introduced an application that translates gestures into voice commands, emphasizing both touch and non-touch interactions. Non-touch gesture recognition, which converts finger movements into commands, allows remote control of devices and avoids touch-based infections. This can be achieved through handheld or wearable sensors or vision-based gesture recognition, facilitating hands-free interaction.

In 1980, Bolt [82] explored natural user interfaces integrating speech and gesture inputs. The study found that combining gestures with speech improved system interpretation accuracy and user experience. It suggested minimizing the need for pauses during speech commands by providing sensory smart prompts to help users understand system processing, thus reducing anxiety.

Billinghurst [83] argued that speech and gestures complement each other, with speech suited for descriptive tasks and gestures for direct manipulation. Most users preferred combining both methods, with 71% using them simultaneously, compared to 13% using only speech and 16% using only gestures. This multimodal approach increased recognition accuracy and reduced command completion time, enhancing user focus on the screen by up to 108%.

The design of user interfaces remains an interdisciplinary challenge. Jalil [84] proposed a framework for “Intelligent User Interfaces” (IUI), shown in Figure 8, aimed at improving performance, emotional engagement, and naturalness in human-computer interactions. This framework employs computer vision and speech recognition technologies to understand natural language. The research highlights that sensing human emotions will be pivotal in future intelligent user interfaces, offering immersive experiences, responsive interactions, and aiding in decision-making and contextual guidance.

Song et al. [85] emphasize that sensor-based interactive interfaces must become more human-centered, with performance characteristics like sensitivity, flexibility, and measurement range being crucial for future applications.

Key points from this literature review include the following.

(1)

Modular design framework: Modular design enhances flexibility and provides a more intuitive interaction experience.

(2)

Design principles: Effective gesture commands often rely on directional sensors to increase interaction naturalness and immersion. Emphasis is placed on paired command combinations and minimizing the use of pause commands during interactions.

(3)

Integration of voice and gesture: The combination of non-contact multi-sensing interfaces (voice and gesture) within the Intelligent User Interfaces (IUI) framework supports the development of an innovative and engaging multi-sensing interactive affective computing system.

Summary details of the multi-sensing interface design principles are provided in Table 5.

Based on the above discussion, it is tried in this study to integrate multi-sensing interactive interfaces with affective computing to explore users’ cognitive emotions. The aim is to create an efficient and natural interaction experience. By analyzing multi-sensing designs and referencing interface design principles, in the study efforts will be made to seek a better design of an interactive system that aligns with these principles. The ultimate goal is to design a suitable affective computing sensor system.

2.3.2. Related Cases of Multi-Sensing Interactive Interfaces

Multi-sensing interfaces overcome the limitations of single-input systems by integrating inputs from multiple devices, enhancing interaction speed, accuracy, and naturalness. This shift from traditional mice, keyboards, and touchpads to simultaneous inputs like speech, text, and gestures increases focusing and immersion. Assistive applications are increasingly important, offering contextual recommendations and improving quality of life in a user-friendly way [86].

In this study, focuses are put on combining natural gesture control with voice recognition. A comparative table (Table 6) has been created to evaluate the characteristics and application areas of six existing cases of multi-sensing interactive interfaces, providing a reference for designing the affective computing sensor system.

2.3.3. A Summary

In smart living applications, integrating multi-sensing inputs is a novel area of human-computer interaction research. In this study, the focus is put on combining “non-contact gesture interfaces” and “voice interfaces” to enhance smart living applications. The key points of the previous literature review are summarized as follows.

(1)

Enhanced accuracy and efficiency: Multi-sensing inputs improve system accuracy and efficiency, leading to greater focusing and immersion.

(2)

User preferences: Users generally favor interfaces that combine non-contact gestures and voice control, with voice interfaces boosting user concentration.

(3)

Intelligent user Interfaces: The IUI framework can enhance performance, emotional engagement, and naturalness in interactions by sensing emotional information.

(4)

Growing importance of assistive features: Assistive features are becoming more critical in multi-sensing interfaces. In this study, it will be tried to explore integrating affective computing into interaction interfaces to improve user quality of life and evaluate user experience in a simple, intuitive manner.

2.4. Concluding Remarks of the Literature Review

In this section, the exploration of emotional mechanisms and behaviors in interpersonal interactions is presented. Key mechanisms from research are summarized, and a review of literature on affective computing and multi-sensing interactive interfaces is conducted, with existing cases analyzed to establish design principles for this study.

The primary aim in this study is to integrate multi-sensing interactive interfaces with affective computing to enhance interpersonal communication. The design principles for this integration are as follows:

(1)

Intentions of interpersonal interaction: Goals include alleviating loneliness, seeking stimulation, satisfying needs, and enhancing self-understanding. In this study, it is aimed to evoke emotional resonance and generate positive experiences by integrating cognitive processes and context, measuring pleasantness, attention, and emotional intensity based on the emotional hourglass model.

(2)

Benefits of affective computing: Affective computing aids in real-time emotional regulation and interpersonal issue resolution, while also enhancing decision-making capabilities.

(3)

Advantages of multi-sensing interactive interfaces: Combining voice and gestures in multi-sensing interfaces surpasses traditional methods, improving focusing and resolving ambiguous inputs through contextual semantic analysis. The interface’s effectiveness is evaluated based on sensitivity, flexibility, and measurement range, enhancing system accuracy and effectiveness.

(4)

Integration and evaluation of sensor data: Integrating multiple sensors to capture user input is assessed using the indicators of effectiveness, simplicity, and pleasantness to highlight the importance of user involvement in the interaction context.

3. Methodology

3.1. Selection of Research Methods

The research methods used in this study are the affinity diagram method, prototype development, indirect observation, and questionnaire surveys. The applications of these methods in this study are illustrated in Figure 9.

The research process began with a literature review to analyze interpersonal interactions, affective computing, and multimodal interactive interfaces, which led to the formulation of the initial design principles for this study. The affinity diagram method was then applied to discuss and refine these principles based on user feedback. A prototype system was developed accordingly and tested through public demonstrations. Finally, the indirect observation and questionnaire survey methods were used to gather insights on user behavior and emotions, which informed further analysis and prototype refinement.

3.2. The Affinity Diagram Method

3.2.1. Idea of Using the Affinity Diagram Method

The affinity diagram method, or KJ Method, introduced by Japanese ethnologist Jiro Kawakita in 1960 [93], is widely used in human-computer interaction research. This technique organizes diverse and unstructured qualitative data related to products, processes, or complex issues by grouping observations based on similarities or dependencies [93,94,95]. Originally applied to synthesize field survey data and generate new hypotheses, the method is now commonly used to analyze contextual inquiry data and guide the design process, providing a qualitative foundation for decision-making [96,97].

The affinity diagram method, utilized primarily in the early stages of the design process, is employed in this study to define and analyze interactive prototypes. Initially, the target user group is identified, and user personas are developed to guide core research needs. Participants then generate descriptive attributes for the target user group on sticky notes, sharing their insights on these users [98,99].

Based on cutting-edge desktop research and literature, six target users utilized the AEIOU framework—focusing on the topics of Activity, Environment, Interaction, Object, and User [100]—to conduct “design thinking” and explore innovative solutions. The clustered data from this process is organized and presented in this study. Qualitative insights, including pain points and core needs of the target user profiles, were identified. Additionally, persona and empathy map design methods were employed to further structure the context and develop detailed user profiles and scenarios, as described in the following sections.

(1) Persona -

The persona concept, introduced by Cooper [101], involves creating virtual characters to represent user groups with specific attributes. Since then, many leading companies have adopted this method. Personas offer a user-centric view, bridging the gap between designers and users. Cooper and Reimann [102] highlighted three key benefits: (1) providing human-centered insights that reduce the designer-user gap, (2) fostering shared understanding and consensus in teams, and (3) enabling precise targeting of user groups, thus improving development efficiency.

(2) Empathy map -

The empathy map, a user-centered design (UCD) method, helps clearly understand user interactions and core needs by empathizing with their situations. It supports software development by validating features through user perspectives [103]. In the design thinking process, the empathy stage involves analyzing stakeholder perspectives to grasp pain points, needs, and expectations. This creates visual clues and application trends based on contextual data [104]. Ferreira et al. [105] noted that empathy maps are well-regarded in software engineering. The map includes six sections: (1) See: what the user observes; (2) Say & Do: user’s verbal expressions and behaviors; (3) Think & Feel: internal experiences and emotions; (4) Hear: environmental influences; (5) Pain: frustrations or difficulties; and (6) Gain: user goals. Empathy maps are effective for depicting and creating target user personas.

3.2.2. Application in This Study and Brainstorming Result

This study utilizes the Affinity Diagram Method to consolidate information from personas and empathy maps, integrating them with the “Job-to-be-Done” (JTBD) framework. The JTBD framework, originating from innovation and entrepreneurship, explores the cognitive psychology and emotional states that drive user behavior. Its goal is to uncover users’ primary motivations and core needs, which is essential for creating designs that are both appropriate and satisfying. By analyzing who (users), what (tasks), how (methods), why (motivation), when (timing), and where (context), the JTBD framework facilitates a thorough contextual investigation. JTBD focuses on three main indicators: (1) Functional: How users perform or approach tasks; (2) Social: The impact of user interactions with others; and (3) Emotional: Users’ feelings and experiences during tasks.

These indicators help in developing a North Star Metric to understand pain points deeply and identify innovative solutions. Overall, comprehending users’ basic needs and core tasks enables the creation of a user-centered design (UCD) experience that meets expectations [106,107,108,109,110,111,112]. In this study, the JTBD framework is used to categorize brainstorming data, while the affinity diagram method consolidates extensive research data for user journey mapping and scenario storytelling.

In this study, six target users formed a co-creation group to brainstorm and use the affinity diagram method for organizing data. This process aimed to gather diverse opinions, ideas, and experiences in a non-judgmental manner, fostering creative thinking and guiding collaborative actions toward consensus. Three core themes emerged from this investigation as described in the following.

(1) Daily life relevance: Ensuring shared understanding among users.

(2) Open and flexible discussion space: Stimulating user inspiration.

(3) Emotional resonance: Providing a comforting and relaxing experience.

The brainstorming results helped shape user profiles and scenarios, focusing on students aged 18 and above. By leveraging these core themes, the study aims to enhance interpersonal communication. The research introduces the theme “what superpowers users wish to have,” allowing users to freely express their opinions or address issues they want to solve with superpowers. The interactive system developed provides a platform for users to express emotions and explore emotional exchanges in interpersonal interactions.

3.3. The Prototype Development Method

Bernard (1984) defined prototype design as a modeling method that efficiently captures user feedback and defines strategies for user needs with minimal time and cost. This method supports iterative refinements, enhancing system development maturity. Hornbæk and Stage [113] highlighted user testing as a common method for evaluating early prototypes, where users interact with the system to provide feedback, guiding design improvements. According to Bernhaupt et al. [114], evaluation testing aims to gather user feedback on usability and experiences, facilitating iterative design refinements. Bernard [115] further noted that this approach improves system output accuracy and aids decision-making, thus supporting systematic prototype development [116].

In this study, five steps based on multi-sensing interactive interfaces and affective computing are taken to develop a prototype system, as described in the following and shown in Figure 10.

(1)

Defining stage - Defining the design needs for the system to be constructed.

(2)

Development stage –

(2.1)

Designing a prototype system based on the techniques of multi-sensing interactive interfaces and affective computing.

(2.2)

Conducting user testing of the prototype system.

(3)

Evaluation stage –

(3.1)

Conducting experiment analysis.

(3.2)

Iterating to Step (2.1) to modify the system whenever necessary.

(4)

Ending stage - Completing the system development

3.4. Indirect Observation Method

Turner [117] highlights that positivist theory, founded by Auguste Comte, emphasizes objective observation as a core research method. Williamson [118] describes observation as both a research method and a data collection technique, while Baker [119] notes its complexity due to the varied roles and techniques required from researchers. Regardless of their involvement level, researchers must uphold objectivity to ensure accurate data collection and analysis. Validity is categorized into descriptive, interpretive, and theoretical types, with theoretical validity assessing how well theoretical explanations align with the data.

Bernhaupt et al. [114] found that objective observation can lead to deeper analysis and iterative system refinements. Future research may leverage observational data (such as images and sounds) or physiological sensing (like eye-tracking, skin conductivity, and EEG) to validate study effectiveness. Advancements in technology enhance indirect observation by improving data recording, communication, and storage, thus providing a flexible and rigorous method for capturing user insights and validating hypotheses [120].

Indirect observation offers the advantage of capturing users’ natural states without the need for researchers to be present for extended periods or intervene directly. This approach yields authentic, reliable data and is suitable for collecting a wide range of information, providing a more objective, comprehensive, and diverse perspective.

Indirect observation is employed in this study as a method for data collection, using natural and non-intrusive sensing of users’ physiological emotional characteristics during system interactions. It involves collecting and analyzing physiological emotional data such as semantic and EEG information. This method not only aids in understanding emotional transmission trends in interpersonal interaction contexts but also strengthens and validates the research.

3.5. Questionnaire Survey Method

3.5.1. Ideas and Applications of the Method

The questionnaire survey method is widely used for its reliability and effectiveness in collecting key data and information [121]. Designing surveys requires clear, straightforward language and avoids ambiguous or leading questions to ensure validity [122]. This method helps explore users’ views and attitudes, guiding subsequent prototype revisions. In this study, a questionnaire will be administered during the public exhibition period, utilizing a five-point Likert scale for evaluation.

The System Usability Scale (SUS), developed by Brooke [123], is a flexible Likert-type scale with ten straightforward items. It assesses user satisfaction with systems based on agreement levels and can be used for minor adjustments or comparative analysis of different systems. The SUS provides a quick measure of usability, focusing on effectiveness and efficiency [124,125].

In human-computer interaction research, Zhang and Adipat [126] identified “effectiveness,” “simplicity,” and “enjoyment” as key usability metrics for evaluating user experience with systems. Davis [27] highlighted a correlation between perceived usefulness and perceived enjoyment in the technology acceptance model. Additionally, Schrepp et al. [34] linked experiences of pleasure, excitement, and novelty to perceived enjoyment, suggesting that emotions can reflect aspects such as system effectiveness, reliability, and interactivity.

Tiger [127] identified four categories of pleasure derived from products: “physiological pleasure,” “social pleasure,” “psychological pleasure,” and “intellectual pleasure,” providing a nuanced framework for understanding different aspects of pleasurable experiences. Jordan [128] emphasized three challenges in studying pleasure: understanding users and their needs, linking product characteristics to pleasurable qualities, and developing quantifiable metrics for measuring pleasure. As shown in Figure 11, Jordan [129] also proposed a hierarchical structure of user needs—functionality, usability, and pleasure—where functionality forms the base, usability is the next level, and pleasure represents the highest level, emerging once usability needs are met.

3.5.2. Purpose of Survey and Question Design

It is aimed in this study to assess through surveys how users enhance interpersonal interactions, contextual awareness, and emotional resonance with the system. Hassenzahl [130] proposed the Hedonic Quality Model, emphasizing that satisfaction arises from subjective experiences of design, with the relationship between usability and hedonic quality influenced by design and context. Hernández-Jorge et al. [131] analyzed emotional communication and found that creating a positive emotional atmosphere, caring about others’ feelings, and active listening are crucial for emotional interaction and communication. Expert validity may be employed to assess the survey’s effectiveness, focusing on content validity. This process involves selecting at least two knowledgeable experts, distributing the survey to them, collecting their feedback, and revising the survey based on their recommendations [132,133].

In summary, the dimensions of “system usability,” “interactive experience evaluation,” and “pleasure experience” are utilized for designing the questions used in the questionnaire survey in this study. The dimensions and questions of each dimension are detailed in Table 7.

4. System Design

In this study, an interactive system called “Emotion Drift” is designed, utilizing multi-sensing interfaces and affective computing. This system explores emotional communication in interpersonal interactions by integrating inputs from various non-contact methods, including gestures and voice, extending beyond traditional input techniques. By combining affective computing technology, the system provides diverse audiovisual feedback based on the user’s emotional state. It offers a user experience that balances pleasure, interactivity, and usability through natural and intuitive interactions. Further details are provided in the following sections.

4.1. The Design Concept of the Proposed System

The COVID-19 pandemic has accelerated the global shift towards virtual social media interactions to minimize physical contact between individuals. As a result, digital interpersonal communication has become a primary mode of exchange in modern times. However, emotions are often conveyed alongside shared text and audio, leading to emotional resonance. To address and mitigate negative emotions that may arise during interactions, the proposed system has been designed with an emotional feedback mechanism, providing visualized outputs that empathize with users’ feelings.

It is aimed in this study to offer timely emotional audiovisual feedback to users and observe emotional rendering phenomena during interactions. The goal is to improve social relationships by alleviating negative emotions through interpersonal sharing and communication, thereby enhancing emotional regulation, empathy, and strengthening relationships. The system “Emotion Drift” is designed to help users achieve these objectives, offering a high-quality and humane experience. An illustration of the proposed system and the performance environment is shown in Figure 12.

The proposed Emotion Drift System constructs a world for asynchronous interpersonal communication, where emotions are thought to flow like “drifting bottles” — rising and falling with the intensity of winds and storms. Users can navigate imaginatively to isolated islands by “rowing” and pick up drifting bottles left by previous users to engage in deeper exchanges and interactions of emotions. These bottles symbolize a safe and private space for users to express their innermost thoughts.

However, because emotional communication should be fluid and historically recorded, the proposed Emotion Drift System is positioned as an emotional perception system. It invites users to enter others’ emotional worlds, sharing voices and expressing feelings together.

4.2. Design of the Multi-Sensing Interaction Process for the Proposed System

4.2.1. Idea of the Design

In this section, the interaction process using the proposed Emotion Drift System is explained. Users can awaken the system screen from standby mode through gestures, creating a pleasant and relaxing environment. While using the system, the user is imagined to be navigating a boat on a calm sea or river. The user controls the direction of the boat by rowing to explore the water space and pick up a drift bottle at indicated locations based on guidance.

Inside the bottle, the user finds four letters left by the previous user, each containing an answer to a question about “becoming a person with superpower.” The current user reads each letter, which is presented in both text and audio formats, and experiences the previous user’s thoughts on the question, as if interacting directly with the previous user. This process is repeated for all four letters.

Additionally, after listening to the previous user’s audio content, the current user can share his/her own thoughts with the drifting bottle, representing the previous user, to express understanding and empathy for the previous user’s feelings. The system also visualizes the current user’s emotional information, measured with an EEG device, creating graphic illustrations and textual meanings for review, thereby offering a user-centered interactive experience.

4.2.2. Detailed Description of the Interaction Process

More specifically, the interactive flow of the system is mainly divided into five stages: the guidance stage, the experience stage, the sharing input stage, the emotional review and output stage, and finally the ending stage, as illustrated by Table 8 and described in detail in the following.

(1) 

Guidance Stage:

(1.1)

System Awakening - The user activates the proposed interactive system by approaching it. Audiovisual effects are used to bring the “emotional” drifting sailing journey to life on the TV screen through gesture control and interaction with objects.

(1.2)

Instruction learning - The user reviews the interactive scene and process instructions.

(2) 

Experience Stage:

(2.1)

Sailing forward - The user rows the boat forward using guided gestures to explore the interactive experience.

(2.2)

Finding a drafting bottle - The user locates a bottle drifting on the water, containing a letter presumably left by the previous user.

(2.3)

Picking up the bottle – Using appropriate gestures, the user retrieves the bottle, which contains four letters. Each letter includes answers to questions about “becoming a person with superpowers,” provided by the previous user.

(2.4)

Taking out a letter – The user imaginatively extracts one of the letters from the drift bottle and displays it on the interface.

(2.5)

Reading the letter content - Through gesture-based operations on the interface, the user reads the letter’s content, listening to the spoken message to experience and empathize with the emotions expressed by the previous user.

(2.6)

Repeating message reading four times – The user repeats Steps (2.4) and (2.5) until all four letters have been read.

(3) 

Sharing Input Stage:

(3.1)

Activating voice recording - The user presses a voice-control button on the interface using gestures to initiate the system’s recording mode.

(3.2)

Answering a question and recording – The user verbally responds to each of the four questions, sharing their emotions interactively with the previous user. The system records these responses for playback.

(3.3)

Transforming the message into text - The recorded voice is automatically converted into text in real time by speech-to-text technology, which is then displayed on the interface.

(4) 

Emotional Review and Output Stage:

(4.1)

Semantic analysis and emotion presentation - Using natural language processing, the system analyzes the text generated in the previous stage along with the brainwave data collected from the EEG. This analysis identifies the user’s emotions and related parameters, which are then displayed on the interface for review, including a text description and four parameter values.

(4.2)

Repeating answer recording and analysis four times – The user repeats the tasks outlined in Steps (3.1) through (3.3) and (4.1) until all four questions are answered.

(5) 

Ending Stage:

(5.1)

Showing the conclusion of emotion analysis – The system presents a summary of the user’s four emotional states through textual descriptions for review.

(5.2)

Drafting a bottle with message letters – The user writes responses to the questions on letters, places them in a bottle, and releases it into the water for the next user.

Table 8. The process of user interaction on the proposed “Emotion Drift System.

Stage	Step theme	User’s interactive action	System’s operation	Interface display or Enviroment situation
(1) Guidance Stage	(1.1) System wakening	The user awakens the system before the Leap Motion by hand waving gesture.	The Leap Motion in the system senses and recognizes the user’s gesture and moves the boat forward
	(1.2) Instruction learning	The user views the interactive scene and the process instruction.	The system displays the process instructions on the TV screen.
(2) Experience Stage	(2.1) Sailing forward	The user sails the boat forward by rowing gesture.	The system shows the rowing action of the user’s two hands and the forward moving of the boat.
	(2.2) Finding a drafting bottle	The user finds out a drafting bottle on the water.	The system displays the bottle in front of the boat.
	(2.3) Picking up the bottle	The user picks up by two-hand gestures the drafting bottle on the water with a letter left inside by the previous user.	The system shows two hands holding the bottle.
	(2.4) Taking out a letter	The user takes out a letter inside the bottle by pushing a button on the interface.	The system displays the cirresponding question and four “push buttons” on the interface shown on the TV screen.
	(2.5) Reading the letter content	The user reads the dsiplayed message on the letter by listening to the message spoken audibly.	The system plays the audio of the message recorded previously whne the previous user used the system.
	(2.6) Repeating message reading four times	The user does the same task of Steps (2.4) and (2.5) until all four letters are read.	The system repeats the operations done in Steps (2.4) and (2.5) four times as requested by the user.
(3) Sharing Input Stage	(3.1) Activating voice recording	The user pushes the “voice-recording button” on the interface by gestures to activate the system’s recording status	The system gets into the voice-recording state.
	(3.2) Answering a question and recording	The user responds by speaking out his/her thoughts to answer a question after pushing a corresponding buttom on the interfce.	The system records the voice of the user’s answer that can be re-played for review if the user push the “listening button.”
	(3.3) Transforming the message into text	The user views the text yielded by the system, which is what he/she said as the answer to the question.	The system transforms the recorded user’s verbal statements into text and displays the result on the interface.
(4) Emotion Review and Output Stage	(4.1) Semantic analysis and emotion presentation	The user views the emotion analysis result on the TV screen, including a text descriprtion and four parameter values.	By natural language processing, the system analyzes the text and brainwave data to identify the user’s emotions and present them on the screen.
	(4.2) Repeating answer recording and analysis four times	The user conducts the tasks of Steps (3.1) through (3.3) and (4.1) repeatedly until all four questions are answered.	The system repeats the operations done in Steps (3.1) through (3.3) and (4.1) four times as requested by the user.
(5) Ending Stage	(5.1) Showing the conclusion of emotion analysis	The user views the concluding textual listing of the system’s analysis of his/her emotion.	The system presents a concluding listing of the textual descriptions of the user’s four types of emotions.
	(5.2) Drafting a bottle with message letters	The user writes imaginatively the answers on letters, puts them into a bottle, and drafts it away for the next user.	The system shows the image of drafting the bottle on the TV display.

Table 8. The process of user interaction on the proposed “Emotion Drift System.

Stage	Step theme	User’s interactive action	System’s operation	Interface display or Enviroment situation
(1) Guidance Stage	(1.1) System wakening	The user awakens the system before the Leap Motion by hand waving gesture.	The Leap Motion in the system senses and recognizes the user’s gesture and moves the boat forward
	(1.2) Instruction learning	The user views the interactive scene and the process instruction.	The system displays the process instructions on the TV screen.
(2) Experience Stage	(2.1) Sailing forward	The user sails the boat forward by rowing gesture.	The system shows the rowing action of the user’s two hands and the forward moving of the boat.
	(2.2) Finding a drafting bottle	The user finds out a drafting bottle on the water.	The system displays the bottle in front of the boat.
	(2.3) Picking up the bottle	The user picks up by two-hand gestures the drafting bottle on the water with a letter left inside by the previous user.	The system shows two hands holding the bottle.
	(2.4) Taking out a letter	The user takes out a letter inside the bottle by pushing a button on the interface.	The system displays the cirresponding question and four “push buttons” on the interface shown on the TV screen.
	(2.5) Reading the letter content	The user reads the dsiplayed message on the letter by listening to the message spoken audibly.	The system plays the audio of the message recorded previously whne the previous user used the system.
	(2.6) Repeating message reading four times	The user does the same task of Steps (2.4) and (2.5) until all four letters are read.	The system repeats the operations done in Steps (2.4) and (2.5) four times as requested by the user.
(3) Sharing Input Stage	(3.1) Activating voice recording	The user pushes the “voice-recording button” on the interface by gestures to activate the system’s recording status	The system gets into the voice-recording state.
	(3.2) Answering a question and recording	The user responds by speaking out his/her thoughts to answer a question after pushing a corresponding buttom on the interfce.	The system records the voice of the user’s answer that can be re-played for review if the user push the “listening button.”
	(3.3) Transforming the message into text	The user views the text yielded by the system, which is what he/she said as the answer to the question.	The system transforms the recorded user’s verbal statements into text and displays the result on the interface.
(4) Emotion Review and Output Stage	(4.1) Semantic analysis and emotion presentation	The user views the emotion analysis result on the TV screen, including a text descriprtion and four parameter values.	By natural language processing, the system analyzes the text and brainwave data to identify the user’s emotions and present them on the screen.
	(4.2) Repeating answer recording and analysis four times	The user conducts the tasks of Steps (3.1) through (3.3) and (4.1) repeatedly until all four questions are answered.	The system repeats the operations done in Steps (3.1) through (3.3) and (4.1) four times as requested by the user.
(5) Ending Stage	(5.1) Showing the conclusion of emotion analysis	The user views the concluding textual listing of the system’s analysis of his/her emotion.	The system presents a concluding listing of the textual descriptions of the user’s four types of emotions.
	(5.2) Drafting a bottle with message letters	The user writes imaginatively the answers on letters, puts them into a bottle, and drafts it away for the next user.	The system shows the image of drafting the bottle on the TV display.

4.2.3. Questions to Answer as Messages for User Interactions

The previously-mentioned four questions to be answered by system users as messages for interactions (see Stage (3) above) are listed in Table 9. The four questions are related as a sequence with continuity in meanings:

Introduction → Development → Turning → Conclusion.

Table 9. Questions to answer as messages for user interactions.

No.	Property	Question content
1	Introduction	What superpower did the previous person share? What are your thoughts on it?
2	Development	What do you think are the good and bad aspects of it?
3	Turning	What superpower would you want and why?
4	Conclusion	How would you recommend this superpower to friends and family?

Table 9. Questions to answer as messages for user interactions.

No.	Property	Question content
1	Introduction	What superpower did the previous person share? What are your thoughts on it?
2	Development	What do you think are the good and bad aspects of it?
3	Turning	What superpower would you want and why?
4	Conclusion	How would you recommend this superpower to friends and family?

4.3. System Architecture and Employed Technologies

4.3.1. Overview of the Architecture

The affective computing functions of the Emotion Drift System rely on gesture and voice recognition for multi-sensing input, and use speech-to-text technology and brainwave analysis for emotion detection and visualization. The architecture for these functions is detailed in Figure 13.

The computational process of the proposed system is divided into three main components:

• Gesture Recognition: The Leap Motion sensor detects hand gestures to control objects within the virtual interface of the Leap Motion software.
• Voice Recognition: The Google Speech API processes voice input, converting speech to text in real time (Speech-to-Text, STT). It also employs natural language processing (NLP) to extract semantic meanings and performs sentiment analysis to assess the emotional content of the user’s speech.
• Brainwave Monitoring: The Mind Sensor EEG headset captures brainwave activity, providing data for observing and visualizing the user’s emotional state.

These components are integrated using the Unity game engine, which provides audiovisual feedback based on detected emotions, as shown in Figure 13. More details will be described in the following sections.

4.3.2. Leap Motion Hand Gesture Sensor

The Leap Motion hand gesture sensor is a device designed specifically for gesture control and hand motion tracking. It uses a combination of infrared LEDs and cameras to detect and track hand movements with high accuracy. The sensor captures hand gestures and movements within a range of approximately 20 to 60 centimeters above the device. The real-time gesture data captured are then converted into information that the computer can interpret, providing users with a natural and intuitive interactive experience.

More specifically, the Leap Motion sensor detects gestures in spatial positions and movement states along three main axes: X, Y, and Z. The horizontal X-axis corresponds to lateral movements; moving the hand left or right changes the X-axis value accordingly. For example, moving the hand to the right increases the X-axis value, while moving it to the left decreases it. The vertical Y-axis is associated with up and down movements. The Z-axis represents depth, relating to forward and backward movements. When the hand moves closer to or farther from the sensor, the Z-axis value changes: moving the hand closer decreases the Z-axis value, and moving it farther away increases it, as illustrated in Figure 14.

In summary, the Leap Motion sensor uses infrared cameras to detect hand gestures in real-time in the physical world (as shown by the green coordinate lines in Figure 15). These gestures are then mapped to virtual world coordinates in the Unity game engine to control interactive objects (as shown by the red coordinate lines in Figure 15).

The system design of this study categorizes gesture commands into five types: “Move Forward (both hands move back and forth),” “Move Left (right hand moves back and forth),” “Move Right (left hand moves back and forth),” “Select Click (extend the index finger to touch the object),” and “Grab Object (open the palm to close the fist towards the object).” These gestures are detailed in Table 10. The development design and judgment logic of the Leap Motion sensor will be further elaborated in later sections.

Obviously, the determination of gesture commands depends on the recognition of the palm shape of the hand, whose skeleton consists of five fingers as shown in Figure 16. By calculating the distance between the fingertip position and the palm position, the current bend of a single finger can be determined. A threshold is set to indicate whether the finger is currently in a straight or bent state. The program developed in this study makes decisions in the aspect in the following way.

(1)

If four fingers on one hand are bent, it indicates a fist gesture.

(2)

If four fingers are straight, it indicates an open-hand gesture.

(3)

By using the codes of the OnTrigger collision event offered by Unity, when the arm is detected to contact with an interactive object, collision detection is activated.

(4)

When the object is grabbable, making a fist will allow the object to be picked up, and an open hand will allow the object to be placed down.

(5)

When the object is an interactive button, the button will be pressed and moved by the arm. This can be cross-referenced with the gesture command table (Table 10) mentioned previously.

Figure 16. Leap Motion hand skeleton description (image source: https://www.ultraleap.com).

Figure 16. Leap Motion hand skeleton description (image source: https://www.ultraleap.com).

4.3.3. Google Speech API

Google Cloud’s NLP technology for sentiment analysis is utilized in this study, employing its open-source machine learning applications. The NLP analysis is performed via a REST API, which supports scalable, high-performance, and reliable communication while being easy to implement and modify. High levels of visualization and cross-platform transferability are provided. Data and service requests are sent to other applications by calling the API. The Google Cloud API service requires JSON data format for cloud requests, with sentiment parameters being extracted through the uploading of voice-related file formats.

The user’s speech is captured using a microphone and encoded for storage. The speech information is then converted into text data via the Google Cloud API service. Sentiment information from the text data is subsequently analyzed using the Natural Language Processing (NLP) technology provided by the same Google Cloud API service, as shown in Figure 17. During this process, NLP techniques are used to analyze the text data to extract two sentiment parameters: score and magnitude. The score reflects the overall emotion expressed in the text, while the magnitude measures the intensity of the emotional content, which is often proportional to the length of the text.

The Natural Language Processing (NLP) technology for sentiment analysis, provided by Google Cloud’s open-source machine learning applications, is utilized in this study. The analysis is conducted using a Representational State Transfer (REST) API architecture, which supports large-scale, high-performance, and reliable communication while being easy to implement and modify. High levels of visualization and cross-platform transferability are supported. The API allows data and service requests to be sent to other applications. The Google Cloud API service requires JSON format for cloud requests, enabling sentiment parameters to be extracted from uploaded voice-related file formats.

The score value (hereafter referred to as the S value) ranges from −1.0 to +1.0, representing the overall emotional tone of the text data from the speech. An S value greater than 0 indicates a predominantly positive emotion, while a score less than 0 indicates a predominantly negative emotion. The magnitude value (hereinafter referred to as the M value) ranges from 0.0 to +∞ (positive infinity) and reflects the quantity of emotional content in the text. The presence of both positive and negative emotional words in the text increases the M value. Table 11 presents examples of text-described emotional states along with their corresponding numerical S and M values, as obtained from the sentiment and semantics analyses shown in Figure 17.

4.3.4. Mind Sensor EEG

In this study, the Mind Sensor lightweight EEG device, as shown in Figure 18, is used to measure and record brain activity. This device is commonly employed in healthcare, wellness, gaming, education, and sports. It operates by using electrodes to detect electrical signals from the brain, reflecting various mental states such as concentration, relaxation, and excitement.

Before starting the interaction with the proposed Emotion Drift System, each participant was required to wear the EEG equipment described earlier. The equipment uses Bluetooth Low Energy (BLE) technology for wireless communication. Once the EEG device connects successfully to Unity, it provides several metrics: the “signal quality value” (denoted as sq) to indicate current signal stability, the “attention value” (denoted as a) to measure the participant’s level of focus, and the “meditation value” (denoted as m) to gauge the participant’s level of relaxation. The system not only records the user’s brainwave states but also quantifies the attention and meditation values, a and m, on a scale from 1 to 100 for emotional visualization purposes.

After experiencing the interaction on the proposed system, the current user will obtain the semantic values M and S as well as the EEG data values m and a. In this study, the EEG device outputs meditation and attention values m and a within the range of 0.0 to 100.0. The formula (1) is used to convert the output value m or a into a scaled conversion value denoted as conversion_x:

$$conversion\_x=(conv\_max-conv\_min)\times {\displaystyle \frac{output}{(output\_max-output\_min)}}-conv\_min$$

(1)

where x = m or a, output_min and output_max represent the minimum and maximum values of the output values, respectively, and conv_max and conv_min denote the maximum and minimum values of the conversion value conversion_x, respectively. By the formula above, the output values m and a of the EEG device can be proportionally converted into the range of –1.0 to +1.0, with the converted meditation and attention values being denoted as conversion_m and conversion_a, respectively.

Furthermore, let $\overrightarrow{OE}$ be the vector formed by the point E at the coordinates (conversion_m and conversion_a) and the origin O at (0, 0). Then, by calculating the angle between $\overrightarrow{OE}$ and the X-axis in an X-Y rectangular coordinate system, one of the 36 angular segments, each spanning 10 degrees, denoted as G, within the 360-degree range that includes the vector $\overrightarrow{OE}$ can be identified. Finally, according to Algorithm 1, G can be used to identify one of 36 possible emotional states, as illustrated in Figure 19. This figure replicates Figure 6 but with the X- and Y-axes renamed to “meditation” and “attention,” respectively, for this study. Specifically, the horizontal axis in Figure 6, originally representing “pleasure,” is now designated as the meditation value m, while the vertical axis, originally representing “activation,” is now the attention value a. It should also be noted that the X-axis and Y-axis in the affect circumplex of Figure 6 have been alternatively considered to represent the parameters “valence” and “arousal,” respectively, in References [52,53].

For example, if E = (0.5, 0.5), the angular direction of the vector $\overrightarrow{OE}$ relative to the X-axis is 45°. According to Step 3 of Algorithm 1, 45/10 = 4.5 is calculated and rounded down to the integer I = 5. This indicates that the 10-degree angular segment containing $\overrightarrow{OE}$ is the 5th segment from the X-axis in the counterclockwise direction, which corresponds to the emotional state “enthusiastic” according to the affect circumplex shown in Figure 19.

Algorithm 1: computing the emotion state from the meditation and attention values.

Step 1: Define two vectors:

- Vector1: Created from the coordinate values (conversion_m, conversion_a) with respective to the origin (0, 0) of the X-Y coordinate system.

- Vector2: A fixed reference vector which is taken to be the X-axis.

Step 2: Calculate the angle between Vector1 and Vector2:

- Use a function to determine the angle θ between Vector1 and Vector2 in the counterclockwise direction with θ to fall in the range of 0 ≤ θ ≤ 360^o.

Step 3: Categorize the angle:

- Divide the angle θ by 10 and round down to the nearest whole number I.

- This integer I represents one of 36 possible 10-degree segments.

Step 4: Decide the output emotion state:

- Find the I-th emotion state counted from the X-axis in Figure 19 in the counterclockwise direction as the desired output.

4.3.5. Visual Displays of System Data and Verbal Descriptions of Users’ Emotion States

As mentioned previously, the Google Cloud API service is used in this study for the analysis of the text data of the user’s speech, resulting in the sentiment-related values called score and magnitude denoted by S and M, respectively. And the Mind Sensor EEG used in this study for the analysis of the user’s brainwaves yields the emotion-related parameter values of attention and meditation, denoted as a and m.

For the purpose of illustrating these data on the interface shown on the TV screen for the user to inspect as shown in Figure 20 which is a copy of the illustration of Step 4.1 in Table 8, the values of these four kinds of parameter data are converted to be in the range of 0.0~1.0, or equivalently, in the range of 0%~100%. Also, four colored scales are employed to indicate the percentages, respectively, with the respective colors being taken to be red, gray, green, and blue, as described in Table 12. Finally, letter papers are also used to show the emotion state of the user, which are of the same colors of the percentage scales.

To illustrate these data on the TV screen interface, as depicted in Figure 20 (a reproduction from Step 4.1 in Table 8), the values of the four types of parameter data are normalized to a range of 0.0 to 1.0, or equivalently, 0% to 100%. Four colored scales—red, gray, green, and blue—are used to represent these percentages, as detailed in Table 12. Additionally, letter papers displaying the user’s emotional state are presented in the same colors as the percentage scales.

Furthermore, the names of the four types of parameter data, namely, score, magnitude, meditation, and attention are renamed as emotion, sensibility, relaxation, and focusing, respectively, for the user to understand more easily.

Finally, both the result yielded by the speech analysis using the Google Cloud NLP API and that yielded by the brainwave analysis using the Mind Sensor EEG are combined to generate a sentence to give a verbal description of the overall emotion state of the user (appearing on top of the four scales in Figure 20), accompanied with a bottle filled with a letter with the color of the emotion scale (“red” for the example shown in Figure 20).

5. System Experience Experiment and Experimental Data Analysis

5.1. System Exhibition and Experiment Processes

An affective computing system called “Emotional Drifting,” has been constructed in this study using multi-sensing interactive interfacing techniques. The system has been publicly showcased in February 2024 with experiments being conducted according to the following procedure.

(1)

Experiment Venue: Design Building II, First Floor, and Interactive Multimedia Design Laboratory in Yunlin University of Science and Technology in Yunlin, Taiwan.

(2)

Participants: Users aged 18 and above.

(3)

Sample Size: 60 participants.

(4)

Procedure: The experiment lasts about 20 minutes per participant. Upon arrival, each participant receives a brief overview of the procedure. They then engage with the interactive system while their behaviors are observed. Finally, participants complete a questionnaire survey. The details from a user’s view are shown in Figure 21.

In this section, the experimental process is documented and explained, with direct observation, questionnaire surveys, and brainwave data (attention and meditation) used as the basis for making post-experience adjustments to the system.

5.2. Indirect Observation and Analysis of Collected Data

5.2.1. Ideas of Applying the Indirect Observation Method

In this study, semantic analysis values and EEG data (attention and meditation scores) were collected from participants during their interactions when using the proposed system. Pearson correlation analysis was used to examine the relationships between the data collected from 60 users, focusing on the distribution of correlations and the linear trends observed. A two-tailed test was conducted to explore associations between the data. Correlations were classified into three levels: low, moderate, and high, as detailed in Table 13.

In addition to examining data relations using Pearson correlation tests, significance levels (p-values) are also computed. A correlation coefficient c with a p-value of 0.05 or less is marked with one asterisk (*) as c* and labeled as “significant” (p < 0.05). When the p-value is 0.01 or less, c is denoted with two asterisks (**) as c** and labeled as “highly significant” (p < 0.01). Finally, when the p-value is 0.001 or less, c is marked with three asterisks (***) as c*** and labeled as “very significant” (p < 0.001). These decision thresholds are detailed in Table 14.

Correlation analysis helps in understanding and explaining the relationships between variables, identifying collinearity, and creating visual linear statistical charts. Some applications of this method in this study are investigated subsequently.

5.2.2. Statistical Analysis of Correlations of Semantic and Brainwave Data

In this section, the emotional data collected by the proposed system, “Emotion Drift,” are examined to explore correlations between different users’ emotional states. Such emotion states are expressed either by the semantic content contained in the users’ messages or by the brainwave data (meditation and attention values) measured of the users, to understand their interrelationships. Such correlation analysis includes investigating the transmission of emotions in message exchanges during interpersonal interactions and the impact of previous users’ emotions on current users.

Note that the semantic content of messages is analyzed by use of the Google Speech API, and the brainwave data are obtained by use of the EEG set, both of the proposed system as mentioned previously. The semantic values range from -1.0 to 1.0, while the brainwave data, including meditation and attention values, range from 0 to 100. Such data from the 60 users invited to use the proposed system were analyzed and visualized in detail.

(A) 

Terminology definitions ---

For convenience of reference, the following terminologies are defined for use in the following discussions. Note that the terminologies are defined from the viewpoint of the current user.

(1)

semantic (listening i) – the semantic data of the i-th message left by the previous user which is listened to by the current user, with i = 1 through 4;

(2)

semantic (speaking i) – the semantic data of the i-th message spoken by the current user, with i = 1 through 4;

(3)

meditation (listening i) – the meditation value measured of the current user when listening to the i-th message left by the previous user;

(4)

attention (listening i) – the same as above except that the parameter value is attention;

(5)

meditation (speaking i) – the meditation value measured of the current user when speaking the i-th message;

(6)

attention (speaking i) – the same as above except that the parameter value is attention;

(B) 

Types of investigated correlations of emotion states ---

The following types of correlations of emotion data are explored in this study.

(1)

Type 1: the correlation between the semantic of the i-th message left by the previous user (and listened by the current user) and the semantic of the j-th message spoken by the current user, i.e., the correlation between semantic (listening i) and semantic (speaking j), i ≠ j.

(2)

Type 2: the correlation between the semantic of the i-th message left by the previous user and the meditation or attention value measured of the current user when listening to the j-th message, i.e., the correlation between semantic (listening i) and meditation (speaking j) or attention (speaking j), i ≠ j.

(3)

Type 3: the correlation between the meditation (or attention) values measured of the current user while listening to the i-th messages and the meditation (or attention) value while listening to the (i+1)-th message, i.e., the correlation between meditation (listening i) (or attention (listening i)) and meditation (listening i+1) (or attention (listening i+1))

(4)

Type 4: the correlation between the meditation (or attention) values measured of the current user while speaking the i-th messages and the meditation (or attention) value while speaking the (i+1)-th message, i.e., the correlation between meditation (speaking i) (or attention (speaking i)) and meditation (speaking i+1) (or attention (speaking i+1)).

(C) 

An illustrated case of emotion correlation ---

As an illustration of how cases of the previously-mentioned types of correlation of emotion states are investigated, the correlation of semantic (listening 2) and semantic (speaking 4), that is, the correlation between the semantic of the i-th message left by the previous user (and listened to by the current user) and the semantic of the j-th message spoken by the current user, is computed using the 60 users’ emotion data left on the proposed system during the system exhibition period.

The Pearson correlation coefficient is 0.266*, with a significance value of 0.040, as shown in Table 15. The average semantic score for the second message shared by the previous users is 0.30 with a standard deviation of 0.36, while the average score for the fourth message shared by the current users is 0.44 with a standard deviation of 0.39, as detailed in Table 16. The linear distribution of these values is illustrated in Figure 22. Despite the overall scattered linear distribution indicating no significant linear correlation, a notable low-level correlation exists between the data. This suggests that more positive semantic analysis scores in the previous user’s second message are associated with more positive semantic feedback in the current user’s fourth message.

(D) 

List of the four types of correlations investigated in this study ---

Totally, fifteen cases of the previously-mentioned four types of correlations have been investigated, with the case dealt with above in (C) as one of them. The results are listed in Table 17.

5.2.3. Concluding Remarks of Statistical Analysis

From the results shown in Table 17, the following facts can be drawn.

(1)

From all the types:

From the cases presented in the table, it is evident that Emotion dataset 1 is positively correlated with Emotion dataset 2, at least at a statistically significant level (p-value < 0.05).

(2)

From Type 1:

Case 1 shows that the more positive the semantic value of the second message shared by the previous user, the more likely the current user is to provide a more positive semantic response when sharing the fourth message.

(3)

From Type 2:

From cases 2 to 4, it is observed that a more positive semantic value expressed by the previous user in the left message is likely to influence the current user to exhibit a more positive emotional trend, with increased meditation or attention in their response.

(4)

From Type 3:

From cases 5 to 9, it is observed that when a user’s brainwave meditation (or attention) value is high while listening to a message, there is a positive correlation trend, with the user’s brainwave meditation (or attention) value remaining high when listening to the subsequent message.

(5)

From Type 4:

From cases 10 to 15, it is observed that when a user’s brainwave meditation (or attention) value is high while sharing a message, there is a positive correlation trend, with the user’s brainwave meditation (or attention) value remaining high when sharing the subsequent message.

5.2.4. Observations of Emotion Changes in Listening and Sharing

In this study, an objective was to investigate changes in the user’s emotions during interactions on the proposed system. As depicted in Figure 19, the two axes of the affect circumplex have been renamed to “meditation” and “attention,” representing relaxation and focusing, respectively, in this study. These axes divide the “emotion space” into four quadrants — enthusiasm, anxiety, depression, and comfort — arranged in a counterclockwise direction.

(A) 

Message listening phases

Analysis of the aforementioned emotion changes during the interaction process reveals that, in the message-listening phases (I) and (II), most users were situated within the enthusiasm quadrant. However, in phases (III) and (IV), users shifted to the comfort quadrant, as shown in Table 18. This transition indicates that, as the phases progressed, users’ emotions evolved from enthusiasm to a more comfortable state. This suggests that using the proposed system to listen to messages from the previous users during interpersonal interactions can elicit both focusing and relaxation states (represented by the enthusiasm and comfort quadrants, respectively) in the user.

(B) 

Message sharing phases

An analysis of the emotional changes during the interaction process reveals that in all four message-sharing phases, most users were within the comfort quadrant, as shown in Table 19. This indicates that using the proposed system for sharing messages in interpersonal interactions tends to induce a state of relaxation (represented by the comfort quadrant) in the user.

5.2.5. Concluding Remarks about Emotion Changes while Using the System

The analysis of emotional changes during message-listening phases (I) and (II) shows that users predominantly experienced enthusiasm. As the interaction progressed into phases (III) and (IV), there was a notable shift to a state of comfort. This transition suggests that the system effectively moves the users from an initial state of enthusiasm to a more relaxed and comfortable state. Thus, the system’s message-listening function can engage the users’ focus initially and then foster relaxation, reflecting its capability to evoke both stimulating and calming emotional responses.

In contrast, during all four message-sharing phases, users consistently remained in the comfort quadrant. This indicates that the system tends to induce a state of relaxation throughout the message-sharing process. Therefore, the system’s message-sharing feature primarily supports a relaxing emotional state, enhancing the users’ comfort during interpersonal interactions.

5.3. Questionnaire Survey and Statisitcal Analysis of Answer Data

5.3.1. Statistics of Participants in the Questionnaire Survey

The basic information of the participants in the questionnaire survey of this study, including gender, age, and experience with online dating apps, was analyzed to show the sample structure. A total of 60 participants with valid data were included, with 63% being female and 37% being male. Among them, 78% had used relevant online dating apps. The statistical summary of the basic information of the research sample is shown in Table 20.

5.3.2. Design of Questionnaire

Based on the questionnaire survey method described previously, the questions for the questionnaire survey conducted in this study are designed according to three dimensions, namely “system usability,” “interactive experience,” and “pleasure experience”, using a five-point Likert scale in which opinions ranging from “strongly disagree,” “disagree,” “normal,” “agree,” to “strongly agree” are assigned scores of 1, 2, 3, 4, and 5, respectively. The questionnaires were distributed to the users for them to fill after they used the proposed system. The collected results are shown statistically in Table 21.

5.3.3. Analysis of Reliability and Validity of Collected Questionnaire Data

Before assessing the system’s effectiveness in terms of three dimensions, “system usability,” “interactive experience,” and “pleasure experience” based on questionnaire responses (Table 21), the reliability and validity of the data (Table 19 and Table 20) must be verified. This verification is performed through a five-step process using IBM SPSS and AMOS, with the steps and results detailed below.

(1)

Step 1: Verification of the adequacy of the answer dataset

To assess the adequacy of the collected answer data, the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity [135] were applied in this study. Typically, a KMO measure value exceeding the threshold of 0.50 is considered to pass the KMO test. And a significance value below the threshold of 0.05 is generally accepted to pass the Bartlett’s test. When both tests are successful, the dataset is considered to be adequately related for subsequent structural analysis.

With the datasets of the three dimensions of questions shown in Table 21 as input to the SPSS program, the KMO measure values and Bartlett’s significant values can computed to be as those in Table 22. It is evident from the table that all the KMO measure values exceed 0.50, and all the significant values for Bartlett’s test are below 0.05. Therefore, both tests have been successfully passed for each dimension of questions, indicating that the datasets from Table 21 are suitable for further structural analysis, as undertaken in the subsequent steps of data verification.

(2)

Step 2: Finding the latent dimensions of the question datasets

In the structural analysis of the answer data of the questionnaire survey, the major aim is to categorize the questions of each dimension into meaningful subsets, each subset being associated with a latent dimension. The exploratory factor analysis (EFA) utilizing principal component analysis, along with the varimax method with Kaiser normalization, was applied for this purpose using the SPSS package. The outcomes with Table 21 as the input are shown in Tables 23 through 25 for the three dimensions, “system usability,” “interactive experience,” and “pleasure experience,” respectively.

After examining Table 23, it is found that the questions in the first dimension “system usability” fall into two groups, RA1 = {A06, A03, A02, A07} and RA2 = {A04, A05, A01}, aligning with two latent dimensions named “functionality” and “effectiveness” in this study according to the common characteristics of the questions in each group. Likewise, from Table 24 and Table 25, similar reasoning can be carried out to group the questions of the second and third dimensions, respectively, and assign names to the resulting latent dimensions. The results, including that for the first dimension, are shown in Table 26.

(3)

Step 3: Verifying the reliability of the answer data

Reliability refers to the consistency of a dataset across multiple repetitions, which can be evaluated using the Cronbach’s α coefficient [136] from the EFA mentioned previously. The reliability of the dataset increases as the Cronbach’s α coefficient approaches 1.0. A dataset is considered reliable when the coefficient exceeds 0.35, and it is judged to have high reliability when the coefficient surpasses 0.70 [137].

With the input data of Table 21, the Cronbach’s α coefficients computed by use of the SPSS for the three dimensions and the seven latent dimensions are detailed in Table 27. It is evident from the table that all Cronbach’s α coefficients exceeds 0.35 and most approach or surpass 0.70, meaning that the collected answer data of both the dimensions and the individual latent dimensions are reliable for further analysis.

(4)

Step 4: Verification of the applicability of the structural model

To establish the validity of the collected answer data, it is necessary to verify at first the appropriateness of the structural model constructed based on the latent dimensions [138]. This can be done by use of the confirmatory factor analysis (CFA) process, utilizing the AMOS package. The outcome of this analysis conducted in this study with the data of Table 28 as the input resulted in three structure-model graphs, as depicted in Figure 23.

It can be observed that the three index values—χ²/df, cfi, and RMSEA—obtained for each dimension are close to their respective threshold ranges: “between 1 and 5,” “greater than 0.9,” and “between 0.05 and 0.08.” This suggests that the structural model, based on the latent dimensions, fits the collected questionnaire data reasonably well, as indicated by Hu and Bentler [139].

(5)

Step 5: Verification of the validity of the question answer data

In Figure 23, which shows the structural model of the questionnaire data, all factor loading values (standardized regression weights) for the paths along the dimensions to the question items exceed or are very close to the threshold of 0.5. This observation suggests that the construct validity of the model is quite good. This validation is further supported by the construct validity values of all latent dimensions calculated through the EFA process, as detailed in Table 29, where it is seen that each value surpasses or is very close to the threshold of 0.6 for confirming construct validity [140].

Through the above five steps of statistical processes, the verification of the reliability and validity of the questionnaire data is completed. Then, it can be proceeded to analyze the questionnaire data of each latent dimension, as conducted in the subsequent discussions.

5.3.4. Detailed Analyses of Answer Data of the Dimension of System Usability

(A) 

Analysis about the latent dimension of “functionality”

The primary purpose of the questions related to this latent dimension, “functionality,” is to explore the functional uses of the proposed system “Emotion Drift” as shown in Table 30, which indicates that the proposed system designed in this study has moderate usability.

(1)

The average values for this latent dimension range from 4.00 to 4.43, with standard deviations between 0.62 and 0.86, indicating user agreement with all question items.

(2)

The average value for question A06, “the various functions of the system are well integrated,” is 4.43 and the standard deviation is 0.59, with over 95% agreement, showing strong consensus on the effectiveness of function integration.

(3)

The average value for question A02, “I am able to understand and become familiar with each function of the system,” is 4.23 and the standard deviation is 0.62, with over 90% agreement, suggesting the system is easy for most users to understand.

(4)

The average value for question A07, “ I would like to use this system frequently,” is 4.00 and the standard deviation is 0.78, with over 80% agreement, but over 15% of the users were neutral, indicating some indifference regarding the willingness to use the system again.

(5)

All four items in this latent dimension have over 80% agreement, reflecting that the users find the proposed system to have good usability.

(B) 

Analysis about the latent dimension of “effectiveness”

The primary purpose of the questions related to this latent dimension, “effectiveness,” is to explore the users’ coherent experience of emotional perception and system processes during interaction as shown in Table 31, which indicates that the users of the proposed system have effective coherent experience in using the system.

(1)

The average values for this dimension range from 4.00 to 4.38, with standard deviations between 0.61 and 0.75, indicating high perceived effectiveness of the system.

(2)

The standard deviation value for question A04, “the emotions output by the system seem accurate,” is above 0.7, showing some variation in user opinions. However, with an average of 4.0, it indicates that most users agree that their perceived emotions align with the system’s results.

(3)

Excluding question A04, each remaining question has over 90% agreement and a standard deviation below 0.62, reflecting a consistent user understanding of the effectiveness of the proposed system.

5.3.5. Detailed Analyses of Answer Data of the Dimension of Interactive Experience

(A) 

Analysis about the latent dimension of “empathy”

The aim of the questions related to this latent dimension, “empathy,” is to explore the extent to which users believe the use of the proposed system “Emotion Drift” fosters empathy-related emotions as shown in Table 32.

(1)

The average values for this latent dimension range from 3.9 to 4.01, with standard deviations between 0.80 and 1.13, indicating that most users have a positive experience regarding the motions related to empathy during the interactions.

(2)

For questions B01, B02, and B04, more than 15% of responses were neutral, indicating notable variability in user opinions. Despite these differences, the overall response still reflects a positive user experience.

(3)

Especially, questions B04 and B01, both about empathy and care about others, have respectively 77% and 79% of agreements or above, meaning that most of the users improved their feelings of these aspects after using the system.

(B) 

Analysis about the latent dimension of “isolation”

The extent of negative feelings experienced by users while using the system is primarily explored in this latent dimension. All question items in this dimension are reverse-coded, and the results are derived by reversing the scores of the negative items in the questionnaire, as shown in Table 33.

(1)

The average values for this dimension range from 3.2 to 4.06, with standard deviations between 0.93 and 1.29, indicating that a positive tendency in user interactions with the system is generally experienced, though with a few neutral responses.

(2)

There are three question items in this latent dimension, all with standard deviations greater than 0.9, and questions B03 and B07 have standard deviations exceeding 1.0, indicating considerable divergence in user opinions for these questions.

(3)

For question B05, “I feel lonely while using the system,” an agreement rate of over 80% was observed, suggesting that the system helps alleviate loneliness and enhances interpersonal relationships.

5.3.6. Detailed Analyses of Answer Data of the Dimension of Pleasure Experience

(A) 

Analysis about the latent dimension of “satisfaction”

The extent to which users experience pleasure and find the proposed system cognitively interesting is primarily explored in this latent dimension. The results of the analysis, presented in Table 34, indicate that good pleasure-inducing qualities are exhibited by the proposed system, which satisfy the users.

(1)

The average values for this latent dimension range from 4.11 to 4.73, with standard deviations between 0.54 and 0.94, reflecting the users’ good experiences of pleasure while using the proposed system.

(2)

For question C04, “I find the system’s interactive experience interesting,” an average value of 4.73 and a standard deviation of 0.54 were recorded, with 99% agreement, indicating that the users are highly satisfied with the interaction process of the system, thinking it to be very interesting and really enjoyable.

(3)

For question C09, “time seems to fly when I am using the system,” the average was 4.11, with a standard deviation of 0.94 and 18% of neutral responses. This indicates that there is discrepancy in user perceptions and thoughts regarding this question.

(4)

Aside from question C09, questions C04 and C05 achieved over 86% agreements, suggesting that most users perceive the system as being able to provide positive emotional experiences like happiness and pleasure.

(B) 

Analysis about the latent dimension of “immersion”

The extent of immersion experienced by the users with the proposed system is primarily explored in this latent dimension. The results of the analysis, presented in Table 35, indicate that moderate pleasure-inducing qualities are exhibited by the proposed system, which comes from the users’ emersion in using the system.

(1)

The average values for this latent dimension range from 4.23 to 4.45, with the standard deviations between 0.67 and 0.88, indicating that positive and pleasant experiences are generally reported by the users.

(2)

For question C07, “I enjoy using the system,” an average value of 4.45 and a standard deviation of 0.67 were observed, with over 93% agreement, indicating that high affirmation of the pleasurable experience provided by the system is given by users.

(3)

For question C09, “time seems to fly when I am using the system,” the average is 4.11, with a standard deviation of 0.94 and 18% of neutral responses. This indicates that some discrepancy exists in user perceptions and thoughts regarding this question.

(4)

This latent dimension includes three questions, with all achieving over 84% agreement, suggesting that the proposed system is generally perceived by the users as providing enjoyable, immersive, and satisfying emotional experiences throughout the interaction process.

(C) 

Analysis about the latent dimension of “relaxation”

The extent of immersion experienced by the users with the proposed system is primarily explored in this latent dimension. The results of the analysis, presented in Table 36, indicate that moderate pleasure-inducing qualities are exhibited by the proposed system, which comes from the users’ relaxation while using the system.

(1)

The average values for this latent dimension range from 3.85 to 4.36, with standard deviations between 0.66 and 0.84, indicating that a positive and active experience regarding stress relief is reported by the users during the interactions.

(2)

Among the three questions in this latent dimension, question C01, “I feel special when using the system,” has a standard deviation of 0.84 with 28% of users providing neutral responses, the highest proportion in this latent dimension. This indicates noticeable variation in users’ opinions regarding this question.

(3)

Questions C02 and C03 have over 85% agreements, indicating that the proposed system is perceived by users as highly effective in providing relaxation and stress relief.

5.3.7. Concluding Remarks about the Analysis of the Questionnaire Survey Results

From the previously presented statistical analysis of the questionnaire survey results, the following conclusions can be drawn.

(1)

In the system usability dimension of the questionnaire, which is divided into two latent dimensions — “functionality” and “effectiveness” — the overall analysis indicates that the Emotion Drift System designed in this study demonstrates moderate usability, as reported by users after interacting with the system.

(2)

In the interactive experience dimension of the questionnaire, which includes the latent dimensions of “empathy” and “isolation,” questions B03, B06, and B07 have standard deviations greater than 1.0, reflecting variability in users’ responses. Nevertheless, the overall results show that the system effectively reduces loneliness, fosters empathy, and enhances interpersonal interactions.

(3)

In the pleasure experience dimension, which is divided into three latent dimensions — “satisfaction,” “immersion,” and “relaxation” — the overall analysis reveals that the Emotion Drift System demonstrates satisfactory pleasure-inducing qualities.

6. Conclusions and Suggestions

In this section, conclusions based on the findings of this study are drawn, followed by some suggestions for future studies are included.

6.1. Conclusions

In this study, the use of multi-sensing interactive interfaces for affective computing to evoke emotional resonance and enhance positive interpersonal interactions has been explored. A comprehensive review of cutting-edge literature and relevant case studies revealed that sharing and exchanging messages in interpersonal interactions conveys emotional information, with pleasure, enjoyment, and interest being crucial for sustaining engagement. The review also highlighted that multimodal interactive interfaces offer flexibility by removing constraints related to form, time, and space, and that systems with emotional functions can recognize human emotions in real-time.

Based on these findings, design principles were developed for creating a system to achieve these goals. A prototype emotion-perception system named “Emotion Drift” was constructed to address negative emotions in social relationships, promote empathetic communication, and provide a higher-quality, more human-centered experience to strengthen interpersonal interactions.

The system was tested through experiments involving 60 participants. Indirect observation and questionnaires were used to assess its effectiveness, and the collected data were statistically analyzed to draw the following conclusions.

(1) 

Emotional transmission in interactions is confirmed, with overall positive user emotions and notable individual variations ?

Emotional transmission in interpersonal interactions is confirmed, with the proposed emotion-perception system “Emotion Drift” identifying 36 distinct emotions. Analysis showed that while users’ overall emotions tended to be positive, there were notable variations in individual emotional experiences.

(2) 

The proposed emotional perception system “Emotion Drift” effectively stimulates positive emotions in users and strengthens interpersonal interactions ?

The incorporation of affective computing technology into interpersonal experiences validated that the system promotes positive, enjoyable emotional resonance among users, leading to high-quality interpersonal relationships.

(3) 

A positive correlation was found to exist between semantic analysis and brainwave metrics of relaxation and concentration ?

Specifically, during the message interaction process, users exhibited relaxed emotions. However, during topic transition to the third message, both the listening and sharing processes effectively increased the user’s concentration and enhanced the quality of positive emotional interaction.

(4) 

The interaction experience based on multimodal and affective computing provides users with pleasure and relaxation emotions ?

Analysis of emotional trends revealed that during listening, brainwave emotions were mainly in the Enthusiasm or Comfort quadrants, while sharing focused more on Comfort. The data confirm a shift from enthusiasm to comfort, demonstrating that the “Emotion Drift” system offers a pleasant and relaxing emotional experience.

6.2. Suggestions for Future Research

Due to the limitations of this study, there are several areas for improvement in the system design. The following directions are suggested for further research:

(1) 

Exploring multi-sensing interaction and affective computing across different age groups?

This study’s sample mainly comprised individuals aged 18-25. Future research should explore emotional transmission trends across different age groups and examine individual users’ emotional data, which this study did not address.

(2) 

Choosing a fixed, quiet, comfortable experimental environment ?

To accurately measure physiological emotional data, experiments should be conducted in a stable, serene, and comfortable environment. This setting will help users relax and exhibit natural emotional responses, leading to more reliable emotional data and improved understanding of emotion transmission.

(3) 

Designing longer induction phases to better cultivate emotions ?

In this study, it was noted that the user’s relaxation feeling increases from the second to the fourth message during the listening phase. To enhance emotional engagement, it is suggested to extend the initial induction phase, giving users more time to fully engage and display emotional trends.

(4) 

Using more intuitive and readable visualizations for emotional information ?

The current system only allows users to infer others’ emotions through voice and context. Future research should explore incorporating more intuitive and proactive visualizations to actively present emotional information to users.

(5) 

Conducting experiments with diverse multi-sensing interaction interfaces and affective computing technologies ?

This study used voice and gesture recognition with Google Cloud semantic analysis and a lightweight EEG sensor. Future researchers should explore diverse interfaces and affective technologies to expand the scope of the study.