1. Introduction
Emojis are colorful ideograms, slightly larger than letters, which can represent facial expressions (or other entities such as hand signs, food, animals, etc…) and are typically used in computer- or mobile-mediated communication, particularly in instant messaging systems and social networks. Emojis may be accompanied by written text and are intended to replace nonverbal communication as much as possible [
1]. They are effectively used in textual communication to compensate for the lack of nonverbal signs and to enrich communication emotionally, promoting the expression of emotions. Some studies, along with the extreme pervasiveness of this modern communication system in messaging, have demonstrated their effectiveness. For example, Aluja and coauthors [
2] measured the acoustic startle reflex modulation during observation of emoji emotional faces and found higher acoustically evoked startle responses when viewing unpleasant emoji and lower responses for pleasant ones, similarly to what obtained with real human faces as emotional stimuli. Again, Cherbonnier and Michinov [
3] compared emotion recognition mediated by different types of facial stimuli: pictures of real faces, emojis, and face drawings and found that, contrary to what one might suppose, participants were more accurate in detecting emotions from emojis than real faces and emoticons. Liao et al. [
4] (2021), comparing ERPs generated by viewing painful and neutral faces and emojis, found that P300 was of greater amplitude in response to faces than emojis, but both types of stimuli elicited late positive potentials (LPP) of greater amplitude in response to negative than positive emotions. The authors concluded that emojis and faces were processed similarly, especially at the later cognitive stages.
Emojis and emoticons have become a worldwide means for expressing emotions in computer-mediated communication, but neuroscientific knowledge regarding how the human brain recognizes them is still scarce. Some studies were performed to compare face and emoticon processing. The results seem to suggest a partial independence in the subserving neural circuits. Yuasa and coworkers [
5] compared cerebral activations during observation of Japanese faces and Japanese emoticons (e.g., (^_^) combinations of vertically organized letters and symbols) through fMRI and found that the face fusiform area (FFA) was active only during processing of human faces (as expected, e.g., [
6]), and not of emoticons. However, the right inferior frontal gyrus, involved in the assessment of stimulus emotional valence, was active during processing of both types of stimuli. This was interpreted as an indication that, even if emoticons were not perceived as faces, they activated higher order social areas devoted to the interpretation of face emotional meaning. Similarly, using emoji stimuli, Chatzichristos and coworkers [
7] failed to find an FFA activation during the perception and matching of positive and negative emojis with words triggering autobiographical memories. Instead, it was found the activation of a neural circuit implicated in emotional and social processing, including the inferior frontal gyrus, the amygdala, and the right temporal pole. However, other neuroimaging studies have found the activation of the FFA and the
Occipital face area (OFA, devoted to face part analysis) during observation of emoticons, which would explain how they would be emotionally recognized and interpreted as facial stimuli [
8-10].
The purpose of this study was manifold. First, it was assessed the communicative effectiveness of sets of emoji and face stimuli through preliminary validation. Then, it was investigated to which extent observers grasped the emotional meaning of real facial expressions and emojis, as a function of stimulus type, valence and complexity. Another important aim of the study was to test the existence of sex differences in the ability to recognize the affective content of faces vs. emojis. Numerous studies have provided evidence that women do better than men in task involving deciphering emotions through facial expressions and nonverbal communication cues [
11-14]. Women would also be more likely than men to express their emotional experiences to others [
15]. This sex difference would be more pronounced with subtle emotions (such as sadness or fear), also would also reflect a greater interest of females in social information [
12] and a more empathic attitude toward suffering of others [
16].
On the other hand, several studies pointed to gender differences in emoji use. Women, in fact, according to some authors, would use emojis more frequently than men [
17]. In line with this evidence, Jones and coworkers [
18] assessed the familiarity and the perceived valence for 70 facial emojis in a large sample of United States college students and found an higher emoji usage and familiarity ratings for women than for men. Since emojis were introduced in the on-line text communication to express sentiments and describe more effectively emotional information, it could be presumed that women would manifest a more intense use of emojis and emoticons than men, similarly to what is observed in real social interactions [
19-20]. However, this does not seem to be the case. Chen and coworkers [
21] documented that 7.02 % of male users used at least one emoji in their typical messages while 7.96% of female were likely to use one or more emojis. Herring & Dainas [
22] reported that females and males social media users interpreted and used similarly emojis in Facebook messages. However, they were surprised to find that male users were significantly more likely to use heart-related emojis than females (female: 17.62%, male: 19.41%, p-value = 0.01). This is contrary to psychological literature where males are reported to be less willing to express love in real life [
23]. According to the authors, such a finding would imply that, although men reserve to express their love in real life, they are more willing to express love through emojis in textual communication. After all, it is possible that social behaviors change as technological contexts change.
To investigate these issues we compared the degree of recognizability of human facial expressions relative to eight different human identities (half male, half female) with that of emojis printed in eight different graphic styles, in a large sample of students. Seven different neutral, positive and negative facial expressions were considered. On the basis of the previous literature it was expected that women outperformed men in the recognition of empathy related, subtle, human facial expressions (e.g., fear) [
24]. In addition, it was generally hypothesized that emoji and face recognizability scores would be roughly comparable, with some occasional advantage for emojis, depending on the specific emotion [
25-26]. The artificiality of this symbolic system would in fact be counterbalanced by emojis’ lack of ambiguity, limited number of fine-grained features, and lack of internal noise related to the variety of human identities.
2. Materials and Methods
2.1. Participants
96 healthy students of local University (48 males and 48 females) aged 18 to 35 years (mean age = 23.89) participated in the study. They were randomly assigned to the facial expression study (48 subjects: 24 males, aged: 23.37; 24 females: 23.37 years) or the emoji recognition study (48 subjects: 24 males aged 23.62; 24 females: 24.91). The current sample size was tested for power analysis by using the program G*Power3.1 for comparing two independent groups with alpha level = 0.05. They were all right-handed, as assessed through the administration of the Edinburgh Handedness Inventory. They all declared to have never suffered from psychiatric or neurological deficits and to have good or corrected-to-good vision. Before taking part in the experiment, participants signed written informed consent forms. The study was carried out in accordance with the relevant guidelines and regulations and was approved by the ethics committee of University of Milano-Bicocca (CRIP, protocol number RM-2021-401). It was conducted online and programmed via Google forms
https://www.google.com/forms.
2.2. Stimuli
56 different human facial expressions and 56 different emoji pictures (i.e. 112 different visual stimuli), depicting 8 variants of 7 universal facial expressions were used in this study. Eight different identities (4 female, 4 male) were used for human faces, while eight different emoji style were used for emoji testing. Stimuli were created and validated as described below.
Emoji. Emojis pictures were drawn from free web platforms (Apple, Google, Microsoft, WhatsApp, Twitter, Facebook and Joypixels and
www.icons8.it) and represented the six basic
Ekman’s emotions [
27]: joy, sadness, surprise, fear, anger, and disgust plus neutrality, in eight different styles. They were matched for size (4 cm of diameter) and average luminance. The original stimuli were slightly modified to adjust their average luminance, color saturation and size.
Figure 1 illustrates some examples of stimuli. Stimuli were preliminarily validated in a behavioral study by means of a test administered via Google Form to select the most easily recognizable emojis among a much larger set of 168 emojis comprising 24 different emoji styles for 7 facial expressions.
Validation was performed on group of 48 students aging on average 23 years. All participants had normal vision, no neurological or psychiatric deficits and possessed diploma, BA or Master degrees. Participants were shown, randomly mixed and once at a time, the various emojis and were required to rapidly observe the picture and decide which, of the seven emotion words typed below, was more appropriate to describe the viewed facial expression, by clicking a check mark within a few seconds. Pictures were displayed at the center of the screen and the experimental session lasted 10 minutes. The eight sets (featuring 56 emojis) associated with the highest accuracy in performance were selected as experimental stimuli. Average hit rate for the final set was 79.40%. In more details, accuracy was 94.53% (SD= 0.51) for joy, 76.04% (SD= 1.85) for surprise, 84.12% (SD= 1.76) for sadness, 96.09% (SD= 2.1) for anger, 57.29% (5.04) for fear, 91.15 % (SD= 0.99) for disgust and 56.51% (SD= 0.99) for neutrality.
Faces. 10 student actors of Caucasian ethnicity were recruited (5 females and 5 males) aging 23 years on average (SD= 1.333) for photos taking. High-resolution pictures of their faces were self- taken with a cell phone at about 40 cm of distance in light controlled conditions, while standing up against a white wall. Actors were required to avoid wearing earrings, glasses, make up, hairpins, pliers, any type of hair embellishments, mustaches, beard. They were also instructed to wear a black t-shirt and gather the hair behind the head. For each of the seven emotions, actors were instructed to imagine a vivid emotional state, while concentrating on a specific autobiographic scenario through the Stanislavsky method, and express it spontaneously. For ‘surprise’ emotion, they were instructed to think of a positive surprise (see
Figure 1 for some examples). Each of the 10 actors provided written consent and filled in the privacy release form.
Stimulus set was validated on a group of 50 students (25 females, 24 males and 1 gender fluid) aging on average 23.7 years (min= 17, max= 34 years). All participants had normal vision, no neurological or psychiatric deficits and possessed diploma, BA or Master degrees. Participants were shown, randomly mixed and once at a time, the 56 pictures relative to the seven facial expressions acted by the eight female and male actors. Subjects were required to rapidly observe the picture and decide which one of the seven emotion words typed below was more appropriate to describe the viewed facial expression, by clicking a check mark within a few seconds. Pictures were displayed at the center of the screen and the experimental session lasted 10 minutes. Overall performance for correctly identifying facial emotions was remarkably high = 87.35% (with a chance rate of 16.7%). No participant performed below an overall rate of 75.0%. In more details, accuracy was 98.47% for joy, 86.73% for surprise, 80.1% for sadness, 89.29% for anger, 72.70% for fear, 85.97% for disgust and 98.21% for neutrality. These recognition rates (in line with the data reported by Carbon [
28]) outperform the accuracy of recognizing facial expressions reported by other studies in the literature (e.g., 57.85% for anger and disgust in Aviezer et al. [
29] and 57.85 for negative emotions in Derntl et al. [
30] thus supporting the qualitative validity of the stimuli. Stimulus set was also evaluated for facial attractiveness by a further group of 12 students (7 females and 5 males) aged between 18 to 25 years. Judges were requested to evaluate the attractiveness of neutral expressions of all identities, by using a 3-point Likert scale, where 1 stood for “not attractive”, 2 for “average” and 3 for “attractive”. The results showed a perfect balance across the two sexes and indicated an “average” degree of attractiveness for the facial stimuli (Females= 1.83; SD= 0.78; Males= 1.82; SD= 0.76). This characteristic of stimuli promotes the generalizability of results to the normally looking population. Face stimuli had the same size (3.37 x 5 cm; 199 x 295 pixels; 3° 22’ x 5°).
2.3. Procedure
The emotion-recognition task consisted in 112 experimental trials, in which participants were first shown a portrait photograph of an adult face (or a facial emoji, according to the experiment) to be inspected for about 2 seconds. The images were equiluminant as assessed by subjecting their luminance values to an analysis of variance (F= 0.099, p= 0.992). Photos of faces and emoji were in color, and were displayed at the center of the screen, on a white background. Immediately below the face or the emoji, there was a list of words (neutrality, happiness, surprise, fear, anger, sadness, disgust), from which they had to select the emotion that they deemed the most appropriate to describe the meaning of the expression. In addition, participants judged how clearly they considered the expression recognizable on a 3-point Likert scale (ranging from ‘1 = not much’ to ‘3 = very much’). The emotion was scored 0 if a different incorrect expression was selected. 5 seconds were allowed for perceiving and responding to the two queries. Participants were instructed to observe one facial stimulus at a time and to respond within 5 seconds, not missing any answer. Only one choice per face/emoji was allowed. The task lasted about 15 minutes.
2.4. Data Analysis
The individual scores obtained from each individual, for each of the 7 facial expressions and stimulation condition, underwent a 3-way repeated-measure ANOVA whose factors of variability were: 1 between-group factor named “sex” (with 2 levels, female and male), 1 between-group factor named “stimulus type” (with 2 levels, emoji and face), and “emotion” between-group factor (with 7 levels, happiness, neutrality, surprise, anger, sadness, fear, disgust). Multiple post-hoc comparisons were performed using Tukey's test. Greenhouse-Geisser correction was applied in case of epsilon <1 and epsilon corrected p value were computed. A non-parametric analysis
(Wilcoxon test) was also applied to all the raw data to measure the distribution of individual scores in relation to the sex of viewer and type of stimulus (regardless of facial expression).
3. Results
The ANOVA performed on recognition rates of emojis vs. faces yielded the significance of Emotion (F 6,552 = 31.575, p˂0.00001). Post-hoc comparisons showed that joy was considered the most recognizable emotion (2.639, SE=0.028) and fear the least recognizable emotion (1.86, SE= 0.07) differing from all the others. The significant interaction of Emotion x Stimulus type (F 6,552 = 13.87, p˂0.00001), and relative post-hoc comparisons showed that recognizability rates were higher for emojis than faces, for all emotions except that for happiness and the neutral expressions (see
Figure 2).
The significant interaction of Emotion x Sex x Stimulus type (F 6,552 = 2.3, p˂0.03), and post-hoc tests showed that women generally outperformed males in the recognition of facial expressions (especially: anger, surprise, sadness and disgust), while males outperformed women in the recognition of all emojis except for fear (see
Figure 3 for means and standard deviations).
Overall, while females were better at recognizing facial expressions than men, especially surprise (p<0.04), anger (p<0.01), and sadness (p<0.01), men were better at recognizing all emojis (especially neutral, surprise and disgust, p<0.01) than women, except for fear emotion, that was better recognized by women.
Wilcoxon tests confirmed this pattern of results showing that, overall, men were significantly better than women at recognizing emojis (z= 5.99, p <0.00001) while women were better than men at recognizing human facial expressions (z= 298, p<0.00001), as shown in
Figure 4. Furthermore, men were better at recognizing emojis than human facial expressions per se (z= 2.87, p <0.004).
4. Discussion
This study compared the recognizability of human facial expressions and emojis, balanced by number of emotions and number of styles/identities tested. Regardless of the sex of viewers, some emojis were recognized more clearly than facial expressions, especially anger. This result is similar to what found by Fischer and Herbert [
25] contrasting facial expressions, emoji and emoticons and finding that emojis and faces were quite good in representing the associated emotions and therefore also in reducing ambiguity.
In addition, for some emotions (i.e., anger) emojis were even better than faces. It should be considered that angry and disgusted emojis, for example, have a slightly different coloring (reddish and greenish), which makes them distinctive. Quite similarly Cherbonnier and Michinov [
26] comparing comprehension of real faces, emojis, and face drawings found that subjects were more accurate in detecting emotions from emojis than from other stimuli, including faces, especially for negative emotions such as disgust and fear.
Overall, in this study, happy emojis were recognized more accurately than others, while fear expressions were recognized with greater uncertainty. These data are consistent with Calvo and Lundqvist [
31] reporting faster reaction times for happy and slowest reaction times for fearful expressions.
Similarly, Fischer and Herbert [
25] found fastest responses for happiness and anger, followed by surprise, sadness, neutral and lastly fear. The same pattern of results was reported for real human faces [
32].
These data are also in agreement with previous literature showing that children with no experience and use of social networks or smartphones recognize joyful and sad emoticons better, while they do worse in the recognition of fear and disgust [
17]. The primacy of happiness expression might be related to the unique physiognomic features (e.g., the typical U-shaped mouth curvature) not present in other emotional expressions [
32] or to the fact that positive expressions are more commonly displayed in social settings (e.g., the social smile), as hypothesized by Schindler and Bublatzky [
33]. The same advantage for recognizing happiness was found with Emoji stimuli, also characterized by the (even more prominent) U-shaped mouth feature for smiling. The emotion of fear, on the other hand, was often confused with other emotions, particularly with surprise, as both were characterized by the arching of eyebrows and the widening of the eyes and mouth.
In this study women outperformed men in the recognition of fearful emojis. This could be partly related to the fact that fearful emojis were more obscure and difficult to be distinguished from surprise, as hits were only 57.29% in validation assessment. Similarly, in a study by Hoffmann et al. [
34] it was found that women were significantly better at recognizing anger, disgust and fear (subtle emotions) than men. According to the authors, this finding was related to the women's greater ability to perceive emotions in a gestalt fashion, making quick and automatic judgements (see also [
35] at this regard). Again, it was found that women were faster than men at recognizing fear, sadness and anger: they were also more accurate than men in recognizing fear [
36]
. In this study, both ANOVA and Wilcoxon tests showed that men were significantly better than women at recognizing emojis while women were better than men at recognizing human facial expressions Quite interestingly, men were better at recognizing emojis than human facial expressions. In general, the women’s ability to recognize human facial expressions more accurately (or faster) than men is a well corroborated notion in psychology and cognitive neuroscience [
20, 37]. Further evidence has demonstrated that women, compared to men, react more strongly when viewing affective stimuli (such as IAPS) involving human beings, thus showing a higher empathic response [
38-40]. Neuroimaging evidence has also been provided that face processing would engage FFA bilaterally in women, and unilaterally (i.e. only over the right hemisphere, rFFA) in men [
41], thus possibly supporting a deeper/more efficient analysis of subtle facial mimicry in females. However, the presumed female superiority in the ability to comprehend emotional nonverbal cues does not seem to include emojis’ processing. Therefore, it is possible that emojis are processed differently, half-way between symbols (such as emoticons or icons) and real faces.
An interesting study by Wolf [
42] observing the effect of gender in the use of emoticons in cyberspace found that to communicate with women in mixed-gender newsgroups men adopt the female standard of expressing more emotions, being therefore not less expressive as with real facial mimicry. The majority of emoticon use by females would however be lying in the meaning category of humor, and including solidarity, support, assertion of positive feelings, and thanks whereas the bulk of male emoticon use would express teasing and sarcasm. This gender difference in used emoji valence is compatible with the present pattern of results, showing how men outperformed women with negative emojis while women were better at deciphering positive emojis such as joyful or even neutral ones.