Prerpints.org logo
Preprint
Article

Beyond Bullying, Aggression, Discrimination, Social Safety: Development of an Integrated Negative Work Behavior Questionnaire (INWBQ)

Altmetrics

Downloads

245

Views

76

Comments

0

A peer-reviewed article of this preprint also exists.

supplementary.zip (1.04MB )

Submitted:

24 May 2023

Posted:

26 May 2023

You are already at the latest version

Alerts
Abstract
Negative work behavior (NWB) threatens employee well-being. There are numerous constructs that reflect NWBs, such as bullying, aggression, and discrimination and they are often examined in isolation of each other, limiting scientific integration of these studies. We aim to contribute to this research field by developing a diagnostic tool with content validity on the full spectrum of NWBs. First, we provide a full description of how we tapped and organized content from 44 existing NWB measurement instruments and 48 studies. Second, we discussed our results with experts (N = 3) in this research field to check for missing studies and to discuss our integration results. This two-stage process yielded a questionnaire measuring physical, material, psychological, sociocultural, and digital NWB. Furthermore, the questions include a range of potential actors of NWB, namely internal (employees, managers) and external actors (clients, customers, public, family members) at work and their roles (i.e., target, perpetrator, perpetrator's assistant, target's defender, outsider, and witness of NWBs). Finally, the questionnaire measures what type of harm is experienced (i.e., bodily, material, mental, and social harm).
Keywords: 
Subject: Business, Economics and Management  -   Human Resources and Organizations

1. Introduction

A broad range of negative work behaviors (NWB) is currently studied by clinicians and researchers, such as victimization (Codina et al., 2020), bullying (Rosta & Aasland, 2018), aggression (Quigley et al., 2020), and deviance (Tabor et al., 2018). Although all these separate studies provide important knowledge about the prevalence and consequences of these negative work behaviors, the limitations of this narrow approach become irrefutably clear.
First, researchers have indicated that various types of NWBs occur simultaneously and sequentially (Einarsen et al., 2011; Gruys & Sackett, 2003; O’Connor et al., 2004; Richman et al., 1999; Rospenda et al., 2009). For instance, swearing (psychological verbal) and hitting (physical attack) often occur simultaneously. Another example of a combination of NWBs is found in #MeToo where traditional types of negative (work) behaviors like physical (rape), material (no promotion) and psychological NWB (gossiping) alternate with digital NWB (pornographic pictures via GSM; Holroyd-Leduc & Straus, 2018). This shows that the real and online worlds also merge or occur simultaneously and sequentially (Black et al., 2012). Further supporting the co-occurrence of NWBs, Privitera and Campbell (2009) found that 5.8% of their respondents reported experiencing only one type of negative act, while 83.5% reported two or more types of real/online negative acts.
Second, next to the often isolated focus on negative work behaviors instead of their co-occurrence, existing measurement instruments usually focus on one actor such as organizational actors (Negative Acts Questionnaire: Revised NAQ-R, Einarsen, Hoel, & Notelaers, 2009) or clients (Healthcare-worker’s Aggressive Behaviour Scale-Users; Ruiz-Hernández et al., 2016), whereas oftentimes multiple actors are involved in NWBs. These actors vary between strangers/public, workers/managers, clients/pupils/customers, and relatives of those as individual or as a group (Merchant & Lundell, 2001).
A disadvantage of measuring actor types separately is that it ignores the increasingly blurring boundaries between internal and external actor types in modern workplaces. Examples of these workplaces are the healthcare (e.g., Cheung, Lee, & Yip, 2018), the police (e.g., Cele, 2018), education (e.g., DeSouza, 2011), or the public sector in general (e.g., Cannavò, La Torre, Sestili, La Torre, & Fioravanti, 2019). In these workplaces, external actors were found to play a key role in employees' interactions with each other, with customers, and actor relatives (e.g., Levine, Taylor, & Best, 2011). Therefore, the dynamics with external actor types such as public, clients, students, customers, providers, and their relatives should be included in research on NWBs to have a complete view on NWBs.
A third limitation of the focus on measuring separate NWBs is that limited insights are gained on the specific nature of its inflicted harm to different actors and their roles. Previous research has shown that NWB inflicts harm to targets (e.g., Verkuil, Atasayi, & Molendijk, 2015), perpetrators (e.g., Heerde & Hemphill, 2019), bystanders as witness (e.g., Sprigg, Niven, Dawson, Farley, & Armitage, 2019), companies (e.g., Porath & Pearson, 2012), and society (e.g., Nielsen, Emberland, & Knardahl, 2017). However, current questionnaires often limit their focus on harm of targets, thus only collecting data of one source (Neall & Tuckey, 2014). Another aspect is that scientists indicated that multiple types of harm such as material (e.g., job loss) and psychological harm (e.g., PTSD) occur in combination (e.g., Campo & Klijn, 2018; Fredericksen & McCorkle, 2013). From these insights, it is important to measure harm among various actors and their roles.
As a final limitation, as these constructs of NWB overlap, separate measurements offer no discriminating validity. This may create the problem that measuring bullying or sexual harassment also has features of interpersonal conflict or aggression, which are considered different types of NWBs. Based on the above observed challenges in measuring NWBs and to further scientific knowledge on this important topic, the purpose of this study is to develop a full spectrum diagnostic instrument to assess Negative Work Behaviors (NWBs). This developmental breadth is searched in the coverage NWB types with co-occurrence patterns, types of harm, actor types and actor roles in accordance with the model of Verschuren et al. (2021). These authors reviewed the NWB field and identified the overlapping and unique aspects of the operationalizations of NWBs to specify a new integrative definition of NWB. The key elements in their definition are: 1) the idea that NWBs are distinguished in five natures (i.e., physical, material, psychological, sociocultural, and/or digital); 2) are associated with different types of harm (i.e., physical, psychological, material, and social); 3) are engaged in by different actors (i.e., criminal/stranger, customer/client/pupil, coworker/manager, or relative); 4) can be performed by individuals, dyads, triads or groups; and 5) are often taking place over time.
Thus, using this model as the guiding framework of this study, fully tapping the concept of NWB requires measurement of the combination of the different NWB types: they can be physical (e.g., kicking), material (e.g., littering), psychological (e.g., isolating), sociocultural (e.g., discriminating on gender), and digital (e.g., hacking), each occurring in different patterns. These patterns of negative work behaviors tend to be systematic (e.g., not a onetime event but a repeated process (e.g., Bayramoğlu & Toksoy, 2017), tend to be ongoing, e.g., occurrence of incidents during a year ( Bennett & Robinson, 2000; Marcus et al., 2002); tend to escalate into more serious NWB e.g., worsens from verbal into physical acts (Nielsen & Knardahl, 2015); and vary from overtly visible such as publicly criticizing (Kaukiainen et al., 2001) to covert and less visible forms as a silent treatment (Thomas & Burk, 2009).
Furthermore, a full measure of NWB should include the different types of harm, namely physical (e.g., cardiovascular disease, Jacob & Kostev, 2017), material (e.g., replacement costs by employee turnover (Porath & Pearson, 2010), psychological (e.g., post-traumatic stress disorder; Baran Tatar & Yuksel, 2018), and social harm (e.g., family consequences; Hoobler & Brass, 2006). Additionally, the NWB measure should include the measurement of different actor types involved in NWBs, i.e., strangers/public (e.g., visitor; Ruiz-Hernández et al., 2016) workers/managers (e.g., teacher; Nagle, 2018) clients/pupils/customers (e.g., supplier; Scholte et al., 2016) and relatives (e.g., friend; Ruiz-Hernández et al., 2016). Finally, the actor roles need to be assessed which requires the measurement of target, perpetrator, and bystander roles as witness, assistant, defender, outsider. This means that a respondent to the NWB questionnaire can have different roles (e.g., target/perpetrator).
To achieve our goal of developing a full spectrum diagnostic instrument to assess NWBs in an integrative way, we thoroughly screened and analyzed existing instruments and studies on digital NWB and its inflicted harm to build subcategories and select measurement items that fit an integrated NWB questionnaire. We also report adjustments that were made to existing items or creating of new items. To validate our efforts, we next report the results of an expert panel who thoroughly evaluated the development of the integrated NWB questionnaire (INWBQ).

2. Method

To arrive at an item pool for an integrated NWB instrument, as indicated above, we used the definition of Verschuren et al. (2021) as a starting point. Using these aspects of NWBs, we examined the literature to identify measurement instruments that can serve as the basis for our INWBQ. Below, we outline the different steps we took to identify these instruments.

2.1. Selection of instruments

Inclusion of construct instruments and studies:
We looked for questionnaires to fill all aspects from the definition of NWB, i.e., type of NWB, harm of NWB, types of actors, actor roles, and occurrence patterns. We searched for these instruments using the 16 most frequently cited NWB constructs as keywords (Verschuren et al., 2021):
1)
workplace aggression i.e., “efforts by individuals to harm others with whom they work, or have worked, or the organizations in which they are currently, or were previously, employed. This harm- doing is intentional and includes psychological as well as physical injury” (p. 38). (Baron & Neuman, 1996);
2)
bullying i.e., “harassing, offending, socially excluding someone or negatively affecting someone’s work. It has to occur repeatedly and regularly (e.g., weekly) and over a period of time (e.g., about six months). Bullying is an escalating process in the course of which the person confronted ends up in an inferior position becoming the target of systematic negative social acts. A conflict cannot be called bullying if the incident is an isolated event or if two parties of approximately equal ‘strength’ are in conflict’’ (p.26) (Einarsen et al., 2020);
3)
mobbing i.e., “situations where a worker, supervisor, or manager is systematically and repeatedly mistreated and victimized by fellow workers, subordinates, or superiors. The term is widely used in situations where repeated aggressive and even violent behavior is directed against an individual over some period of time.” (p.379) (Einarsen, 2000);
4)
harassment/discrimination i.e., “interpersonal behavior aimed at intentionally harming another employee in the workplace” (p.998) (Bowling & Beehr, 2006);
5)
workplace deviance i.e., “voluntary behavior that violates significant organizational norms and, in so doing, threatens the well-being of the organization or its members, or both” (p.555) ( Robinson & Bennett, 1995);
6)
counterproductive work behavior i.e., “volitional acts that harm or are intended to harm organizations or people in organizations” (p. 151) (Spector & Fox, 2005);
7)
workplace violence i.e., “the act or threat of violence, ranging from verbal abuse to physical assaults directed toward persons at work or on duty. The impact of workplace violence can range from psychological issues to physical injury, or even death” (p.1) (NIOSH, 2023);
8)
abuse i.e., “interactions between organizational members that are repeated hostile verbal and nonverbal, often nonphysical behaviors directed at a person(s) such that the target’s sense of him/herself as a competent worker and person is negatively affected” (p.212) (Keashly, 2001)
9)
terror i.e., "hostile and unethical communication which is directed in a systematic way by one or a number of persons mainly toward one individual. These actions take place often (almost every day) and over a long period (at least for six months) and, because of this frequency and duration, result in considerable psychic, psychosomatic and social misery" (p.120) (Leymann, 1990).
10)
injustice i,e., violating distributive, procedural, interpersonal rules (Colquitt & Rodell, 2015).- Distributive injustice i.e., “ dissatisfaction and low morale related to a person’s suffering injustice in social exchanges… felt injustice is a response to a discrepancy between what is perceived to be and what is perceived should be (e.g., effort and reward, allocation of scarce rewards between different people)” (p. 272) (Adams, 1965);- Procedural justice i.e., “members’ sense of the moral propriety of how they are treated—is the “glue” that allows people to work together effectively. Justice defines the very essence of individuals’ relationship to employers. (p.34). In contrast, injustice (e.g., inconsistent treatment, discrimination or ill-treatment, imprecision, ethical flaw, or prejudice), is like a corrosive solvent that can dissolve bonds within the community. Injustice is hurtful to individuals and harmful to organizations” (p. 36) (Cropanzano et al., 2007); -.Interpersonal Justice i.e., “treating an employee with dignity, courtesy, and respect” This includes “informational justice: sharing relevant information with employees.” (p.36) (Cropanzano et al., 2007).
11)
interpersonal conflict i.e., “range from minor disagreements between coworkers to physical assaults on others. The conflict may be overt (e.g., being rude to a coworker) or may be covert (e.g., spreading rumors about a coworker)” (p.357) (Spector & Jex, 1998);
12)
victimization i.e., “an employee’s perception of having been the target, either momentarily or over time, of emotionally, psychologically, or physically injurious actions by another organizational member with whom the target has an ongoing relationship” (p.1023) (Aquino & Lamertz, 2004); -scapegoating: i.e., “an extreme form of prejudice in which an outgroup is unfairly blamed for having intentionally caused an ingroup’s misfortunes” (p.244) (Glick, 2005);
13)
micropolitics i.e.,” referring to employees’ perceptions in organizations, they often describe political behaviors in negative terms and associate these with self-serving behaviors, usually at the expense of others” (p.139) (Poon, 2003);
14)
ostracism i.e., “the exclusion, rejection, or ignoring of an individual (or group) by another individual (or group) that hinders one’s ability to establish or maintain positive interpersonal relationships, work-related success, or favorable reputation within one’s place of work” (p.217) (Hitlan et al., 2006);
15)
incivility i.e., “low-intensity deviant behavior with ambiguous intent to harm the target, in violation of workplace norms for mutual respect. Uncivil behaviors are characteristically rude and discourteous, displaying a lack of regard for others” (p.457) (Andersson & Pearson, 1999);
16)
social safety i.e., “in a systemic or socio-ecological approach (Barboza et al., 2009; Ferrer et al., 2011)), the dyad (victim and perpetrator), the triad (dyad and bystander) and the group are placed in larger systems around them such as the school, neighborhood and society. Therefore, power relations are influenced by the cultural and social context (Carrera et al., 2011)” (p.109) (Broersen et al., 2015)
We focused on the existing published measurement instruments of these constructs, preferably validated ones. This search yielded a total of 34 construct instruments. In the expert round we added 10, mainly unnamed, validated instruments. This resulted in total 44 instruments of which 38 were validated. However, we also acknowledged that existing measures of NWB might lack aspects that are becoming increasingly important in modern organizations (i.e., cyber-enabled and/or cyber-dependent digital NWB; Furnell, 2002) or that are part of our definition of NWB but not present in existing measures of NWB (i.e., material organizational damage; Giga et al., 2008). Inclusion of this missing information yielded a total of 16 studies on digital NWB, and 32 studies on harm that provided input to the development of the INWBQ. We listed total 44 instruments and 48 studies in Table 1, including the goal, number of cyber enabled/dependent NWB items, questions to actor types and actor roles, number of items on harm, rating scale & anchors, duration, total number of items, and validity. The added instruments in the expert round were indicated with an Asterix. We will further unpack the content of Table 1 in the Results section.
Several measurement instruments were excluded from our effort to integrate NWB instruments because their items were derived from included instruments in this study (e.g., Luxembourg Workplace Mobbing Scale (LWMS), Steffgen et al., 2019; Gutenburg Health Study (GHS), Garthus-Niegel et al., 2016; Escala de Abuso Psicológico Aplicado en el Lugar de Trabajo Revised (EAPT-T-R), Escartín et al., 2017; Short Negative Acts Questionaire (SNAQ), Notelaers et al., 2019), because they were not work-related (e.g., Contextual Victimization by Community Violence; Gurrola-Pena et al., 2018), or because they addressed coping behaviors rather than NWBs (De-escalating Aggressive Behavior Scale; Mavandadi et al., 2016). (see Table 1, Supplementary Materials).

2.2. Description of steps in developing the INWBQ

Coding decisions

From the definition of NWB presented above, we coded the dimensions. To distinguish the NWBs and harm dimensions, we formulated physical harm into bodily harm, material harm into material damage, and psychological harm into mental harm. Because we wanted to develop a questionnaire that is broadly applicable to sectors, we chose for the broad group of actor types and coded these dimensions after the model of Merchant and Lundell (2001) in stranger, coworker, customer/client/pupil, relative. The dimensions for actor roles were defined as perpetrator, target, assistant, outsider, defender as in the Bullying Participants Behavior Questionnaire (BPBQ; Demaray et al., 2016). (See Table 2)
We deleted the dimension content that was too general, unclearly worded, or not work-related, combined similar ones, and divided the remaining content into subcategories. Then we ranked the amount of NWBs of each subcategory. We selected items for each dimension using this ranking.
To include the content of two instruments, the Injustice Scale (Khattak et al., 2021) and the Perception of Organizational Politics Scale (Kacmar & Carlson, 1997) we reformulated their 24 reverse-scored items into NWBs. This is because these reverse-scored items measure positive rather than negative work behavior.
NWB is a systematically repeating pattern, over a longer period, that can escalate from mild to serious forms (Leymann, 1996; Nielsen & Knardahl, 2015), and occur from covert without being visible, to overt and visible (Khan et al., 2014; Neuman & Baron, 1998), for several months to a year (Baran Tatar & Yuksel, 2018; Einarsen & Skogstad, 1996). We analyzed the measurement of these patterns (i.e., systematic frequency, duration, escalation, visibility) in the instruments and merged these modes for adoption in our questionnaire.
Finally, after the coding process was finished, we adjusted the questionnaire on five points (i.e., removing double barreled parts of selected items, collecting anonymous digital NWB under the covert NWB, asking concrete behaviors, questioning harm, asking mixt participant roles, develop items on escalation and visibility, adjusting items to external actors). Next, we presented our INWBQ to a panel of experts in this research field. Based on their comments and recommendations, final adjustments were made. (See Figure 1)

3. Results

3.1. Inventory of instruments

In Table 1 we listed the selected instruments and studies. The abbreviations of the unnamed instruments were taken from the study of (Escartín et al., 2021). The goals of the questionnaires could be divided into three types: development (e.g., valid and reliable measure to assess cyberbullying; Farley et al., 2016), prevalence (e.g., identify and address workplace violence in the health sector of nine countries; Di Martino, 2009), and research on a causal relationship of coping or harm with NWB (e.g., on job satisfaction, intention to quit, organizational commitment and deviance; Lim & Teo, 2009).

3.2. Dimensions and item selection for NWB, Harm, Actor types & roles

3.2.1. Dimensions NWB

The NWB dimensions were physical, material, psychological, sociocultural, and digital. We filled these dimensions with NWBs from the instrument items and formed subcategories. For instance, for the dimension material NWB these were property, resources, job decisions, sabotage, and displays of NWB content. Then, we reduced the number of NWBs[1] by combining similar ones (e.g., waste and litter). Double barreled items were distributed separately among the dimensions and subcategories (e.g., causing damage with financial costs were divided among material NWB and material harm. Next, we selected items from the instruments for each final subcategory. For example, for the dimension material, we selected for the subcategory material property the item ‘Somebody purposely dirtied someone’s workplace. (CWB-C), and for the subcategory material displays the item ‘Someone distributed dirty pictures or stories (SEQ). (See Table 3. with NWBs of double barreled and multiple-choice items separately included in red)
In the dimension digital NWB, we made two subcategories reflecting both cyber-enabled NWB and cyber-dependent NWB because they differ. Cyber-enabled NWB concerned traditional NWBs also using a digital medium such as a GSM. Cyber-dependent NWB cannot occur without a digital medium such as identity theft. Cyber in these two subcategories is another term for digital, derived from the study of Furnell (2002). (See Table 3 from Supplementary Materials).

3.2.2. Dimensions Harm

The dimensions of harm that we wanted to cover with the measure, were bodily harm, material damage, mental harm, and social harm (Verschuren et al., 2021). NWB has been conceived as behavior that harms or threatens to harm others at work (e.g., workplace aggression, Baron & Neuman, 1996; bullying, Namie & Namie, 2000; workplace harassment, Bowling & Beehr, 2006; Counterproductive Work Behavior, Sackett & DeVore, 2001; workplace victimization, Aquino & Lamertz, 2004). The NWB instruments ask several types of inflicted harm by NWB such as bodily harm (e.g., physical complaints, SVI), mental harm (e.g., fear, Cyber aggression). On the other hand, various studies used separate questionnaires to show that NWB is associated with bodily harm i.e., sleep difficulties (Hansen et al., 2014), health impairment (Elovainio et al., 2002), cardio vascular disease (Kivimaki et al., 2003), pain conditions (Takaki et al., 2013). (See Table 1. for studies on harm).
These studies also showed that NWB inflicts temporarily mental harm i.e. psychological distress and stress reactions (e.g., Hogh et al., 2012); lower job satisfaction (e.g., Arenas et al., 2015), and persistent mental complaints i.e., depression, anxiety, burnout and PTSD (see Verkuil et al., 2015 for a review); turnover intention (e.g., Mathisen et al., 2008). Moreover, it has been shown that these outcomes result in material harm i.e. sickness absence (e.g., Elovainio et al., 2002), and social harm i.e., to family members (e.g., Hoobler & Brass, 2006; Thompson et al., 2020; Zivnuska et al., 2020). (See Table 1. for studies on harm)
Because four dimensions of harm were part of our integrated NWB definition and some types of harm were not found in the instruments, we included harms from the studies to fill these dimensions and form subcategories. For instance, for the dimension mental harm these were temporarily mental complaints, persistent mental complaints, and coping. We reduced the number of harms by removing subcategories that were too general (e.g., negative psychological outcome), and by combining similar harms (e.g., fear, scared, anxiety). Then selected items for each subcategory. For example, we selected the item 'Did the NWB cause any of the following temporarily mental complaints’ with the multiple-choice answers ‘felt anxious, embarrassed, stressed out, angry, affect social contacts, had difficulties concentrating’. (See Table 4 from Supplementary Materials).

3.2.3. Dimension actor types

The subcategories for the dimension actor types were criminal/stranger, customer/client/pupil, coworker/manager, or relative (Merchant & Lundell, 2001). We gathered the actor types of the instruments for each subcategory such as internet user for stranger (cyber fraud). We added a subcategory for actor group of people based on the content of the instruments (e.g., citizens, shared interest group, organizational group, POPS). After deleting content in subcategories that was too general (e.g., stranger: individual), and combining different names for the same actor types (e.g., worker, subordinate, employee, colleague combined into coworker), we obtained our dimension with subcategories for actor types (See Table 5).
Although we found several items to capture actor types, we chose to select prior to the study which actor types are relevant to be selected for sampling. Of importance here is to delimit what actor types are considered part of an organization’s network. For instance, this can be done through a realist approach in which organizations themselves determine what the social boundaries are, or a nominal approach in which the boundary is conceptually imposed from the analysis goal (Laumann et al., 1983).

3.2.4. Dimension actor roles

Most of the instruments (37) question actors in the role of target (e.g., Being given tasks with unreasonable deadlines, NAQ-R). Other instruments (11) question actors as perpetrator (e.g., Acted rudely toward someone at work, Interpersonal and Organizational Deviance scale). A minority (5) question bystanders as witness (e.g., People in this organization attempt to build themselves up by tearing others down, POPS).
The Bullying Participant Behavior Questionnaire (BPBQ, Demaray et al., 2016) question three more bystander roles: assistant, defender and outsider. The assistant is willing to encourage, join in, or aid a bully in bullying others (e.g., I have made fun of someone when they were pushed, punched, or slapped). The defender participates in behaviors related to defending on behalf of a victim (e.g., I tried to make people stop spreading rumors about others). The outsider acknowledges bullying occurs but chooses to actively ignore it. (e.g., I pretended not to notice when things were taken or stolen from another student).
We followed the example of the BPBQ and added the role of witness. In our questionnaire, we combined subcategories referring to bully, aggressor, attacker, perpetrator into perpetrator, and combined subcategories of victim and target into target (See Table 5). In section 3.3.6 we describe how we adjusted our items to the actor roles.

3.2.5. Occurrence patterns

The NWB dimension that reflects occurrence patterns was subcategorized as systematic frequency, duration, escalation and visibility based on the definition of Verschuren et al., (2021). Below we discuss each pattern with our choice for measurement.

Systematic

Most NWB instruments (23) measure the systematic repetitive pattern of NWB as a frequency of occurrence using a 5-point Likert scale. We chose this 5-point scale to leave a center point as an option for a neutral answer (Robinson, 2018). The 3- and 7-point scales also offer this possibility. However, the 3-point has too few options which can make the choice feel forced, while the 7-point offers so many options that nuances fall away, and disengagement behavior of respondents increases (Bougie & Sekaran, 2020). We deleted the scales without frequency (e.g., disagree- strongly agree). We combined the 5-point anchors for the INWBQ into: never, rarely, at least once a month, at least once a week, daily.

Duration

The measurement instruments use different terms for duration. We summarized this as last year at work. Reporting periods greater than 12 months can lead to potentially serious biases in self-reports due to memory distortions (Arvey & Cavanaugh, 1995; Dalal, 2005).
However, when we only ask about the frequency of digital NWB (e.g., weekly), we do not collect information about the weekly amount of use on a medium. By charting the hours of use per day, indicating weekly digital NWB, we have more information on this.
The inventory of place and time of digital use is what Alhabash et al., (2013) called the viral reach of users. This refers to the interactivity of users on digital media with messages distributed and shared online. We adopted this term for the subcategory medium use with multiple-choice questions on the type of medium (e.g., Cloud storage) and duration of medium use (e.g., 1/2-1 hour a day; Gofin & Avitzour, 2012). We preferred to ask hours of use, a percentage usage per day seemed too abstract and difficult to answer.

Escalation

As Escalation is an essential pattern of all NWB labels, it is remarkable that we found only 2 items about escalation: 'Not reporting a problem so it would get worse' (CWB-C, Fox et al., 2001) and ‘escalation from online NWB to offline NWBs’ (Cyberstalking, Every-Palmer et al., 2015). Because these two items were quite specific on reporting and online NWB, we developed two general extra items. See 3.8 under Adjustments.

Visibility

To include items on visibility is important because these tactics have a function disguising identity of the perpetrator (Baron & Neuman, 1996; Kaukiainen et al., 2001), and in maximizing harm while minimizing danger to perpetrators, also indicated with the “effect-danger ratio” (Björkqvist et al., 1994). An example of this is the invisible role of the instigator in digital NWB (Henrichsen et al., 2015). Another more positive reason for invisible NWB is to keep an option for restoration (Skarlicki & Folger, 1997).
However, because most items in the instruments were covert, overt NWB could not be adequately measured. Therefore, following the example of Rivera and Abuín (LIPT, 2003), we strived for about one-third of the items to be overt/visible NWB. We also formulated two extra questions on visibility. See section 3.3.6 for this adjustment.

3.3.6. Adjustments

After establishing dimensions with subcategories for the INWBQ, and selecting items for each of these subcategories, we adjusted some of these items. Our first adjustment involved removing double barreled parts in the selected items. These are confusing to the respondent and cannot be scored separately. For example, we removed ethnic in the item " Used racial or ethnic slurs to describe you” (FSS).
Another adjustment concerned the anonymous nature of digital NWB. Although anonymous behavior may be visible, we collect it under the visibility pattern covert NWB. This is because we consider an unknown perpetrator as covert NWB.
We decided to ask no constructs in the items (i.e. Verbally abused someone at work, CWB-C). A prior explanation may not preclude interpretations by the respondent. Therefore, we chose for concrete and objective behaviors, and we deleted all construct names in our reduction process of items. In this line of reasoning, our third adjustment concerned our way of asking harm with the construct names as depression, burnout, and PTSD.
These inflicted types of harm by NWB were frequently examined with separate questionnaires (e.g., BDI for depression Beck et al., 1996); IES for PTSD Weiss et al., 1997). These questionnaires contain questions on symptoms of dimensions within the relevant constructs. Several NWB questionnaires from this study (e.g., violence research) ask burn-out, depression or PTSS inflicted by NWB with one single question. We considered it inappropriate for respondents to self-assess their mental illness. Instead, in three items we ask them a combination of symptoms of depression, PTSD, and burnout without naming this a mental illness.
More specifically, for the NWB-inflicted harm depression, we ask a combination of negative mood (Thurston et al., 2019; Verkuil et al., 2015; Zivnuska et al., 2020), loss of interest (SVI), withdrawal (Slitter 2012), feeling worthless (Cyber aggression), feeling embarrassed (GWHQ), and sleep problems (WCM). In doing so, we align with the measurement of depression symptoms (Beck et al., 1996). For the NWB-inflicted harm burnout we ask a combination of emotional exhaustion (Hyde et al., 2006; Verkuil et al., 2015), changed work attitudes (e.g., cynic, less professional) (Giga et al., 2008; Porath & Pearson, 2010), little self-esteem (JOS), alienation (SVI). To make respondents understand this last symptom we reworded alienation in “not as the one I used to be”. For the NWB-inflicted harm PTSD we ask a combination of repeated thinking (SVI), avoiding (FSS), dreams and images of past events (Cyber aggression), blame myself (Cyber harassment), emotional reactions as anger (Cyber aggression), to align with the measurement of PTSD symptoms (Weiss et al., 1997). To make respondents understand the question we reworded repeated thinking as flashbacks and avoiding as avoid places-people-activities.
The fourth adjustment concerned asking about mix actor roles in NWB. To this end, we rewrote all NWB items to the most neutral role of witness (changes in items for this role adjustment were underlined). Escartín et al. (2009) showed that different degrees of severity can be asked regardless of the participant's role as target or witness of NWB. This makes the witness role suitable for interrogating NWB frequencies. Also, this role provides a good starting point to ask about other roles the respondent fulfilled. For this, we ask the respondent what behavior he/she performed in the situation in question, with multiple options as a possibility to tick. For instance, ‘someone received unreasonable work demands’, first answer: frequency: 1. never – 5. daily, second answer: tick all that apply: I looked away, I encouraged it, I openly disapproved it, etc.
Also in Part II, we ask the respondent from his role as a witness to the complaints experienced by colleagues and the organization inflicted by NWB. Because this method of questioning could not be applied to the symptoms of depression, PTSD, and burnout, we made the only exception here. For these specific types of harm, we ask whether the respondent experienced these combinations of symptoms themselves.
The fifth adjustment concerned the occurrence patterns escalation and visibility. As we found only two items on escalation in the instruments, we formulated two additional questions for this pattern. For example: “Did the behavior escalate from mild into more serious or physical types over time?” As we found one item on visibility, we formulated two extra items for visibility. For example: “Did the negative behavior change from covert into a more overt and visible nature?” Furthermore, we selected one-third overt items for visibility, a total of 25 overt (indicated in Italics) and 42 covert items. Although the covert items are observable by many potential witnesses in the work setting, mental elements as ‘trivial’ ‘purposely’ ‘unfair’ or ‘necessary’ are not observable and therefore we considered them as covert NWB.
Our final adjustment concerned the formulation for external actor types. In this way these questions could be answered by external actor types. For instance, we changed “salary” into “payment”.

3.3.7. Expert panel

We asked the expert panel if we were missing important instruments and studies, if they would add additional steps to our method, to consider with us the way of asking actor roles, and their opinion on the results. Three experts in the field, Michelle Demaray, Jorge Escartin Solanelles, and Kara Ng[2], provided answers to these questions. Their valuable additions in literature and studies allowed us to add 10 important instruments (see Table 1). Some suggested short scales, used in a variety of national cohort studies, were not included. This is because they contain items from instruments we already included.
Another point was the need to clearly articulate the INWBQ’s practical use in the introduction. How do we want the scale to be used? The purpose of our questionnaire was two-fold: to measure according to the integral definition of NWB, and to provide organizations with tools to undertake interventions.
One of the experts pointed out that recreating an essentially longer scale of all NWBs and roles may be a barrier to researchers. If researchers are supposed to pick relevant subscales, why wouldn’t they just pick an established scale like in Table 1? That is of course always possible, but the overlap between instruments weakens the results of such research. In other words, the question remains whether e.g., withdrawal is only a consequence of Incivility, measured with the WIS (Sliter et al., 2012). Because this instrument overlaps with other construct instruments, these results remain obscure. To clarify the overlap between instruments, we decided to conduct a data analysis of the overlap between the items of the instruments in this study. Because this was not the purpose of this study, we present this research separately. Furthermore, to substantiate the item selection per dimension we made a ranking of NWBs in the different subcategories. This substantiation can help researchers when considering their choice of an instrument. These two additions also respond to another expert's request to make a clearer case for why the INWBQ with this breadth, including digital/online behavior, is necessary.
Our steps in the method to develop data, subcategories, design, and items seemed thorough and clearly delineated to the experts. The empirical and theoretical arguments for this method were found to be adequate. Our rationales were both: empirical by this study, and theoretical based on our review and integrated definition.
Furthermore, the experts really challenged us on our way of asking the different actor roles. For instance, as each subcategory should have items for each role type this would create a very long list to complete. Suggested was to lessen the number of roles to perpetrator, victim, and bystander and within these roles various types of NWB (e.g., physical, verbal). Another expert rightly pointed out that these roles are not fixed but mixed (Ireland, 2013). For instance, research indicated that more than one half of the bullies also reported being victimized (Haynie et al., 2001).
Because we determined to stick to our original definition, we formulated the questions in the witness role. Choosing the witness role prevents respondents from viewing themselves in a favorable light, not wanting to disclose active involvement as a bully or exaggerating their involvement as a target (Juvonen & Graham, 2016). Subsequently, for each item, we ask which of the five roles the respondent fulfilled also. We question these roles with concrete behavior per role.
One expert explained us that in bullying there is also ‘uninvolved’ youth. Therefore, there is not much difference between witness and outsider because some kids say they don’t see it (Haynie et al., 2001). As this could also be the case for adults, we chose two very clear behaviors for the outsider role as I shut my door, I looked away. Furthermore, it was unclear what instructions would be presented to the person completing the measure. We included an introduction for the INWBQ.
Another important expert addition was about measuring behaviors and consequences in the same questionnaire. In other fields, questionnaires only on consequences have been created independently. As harm is an essential part of the most frequently cited NWB definitions (see our method for these) we included it. Moreover, combining NWB and its potential harm can help organizations understand the relationship between the risk and its damage, enabling effective risk management, resource allocation, decision-making, and compliance. By prioritizing risks based on their potential impact, organizations can allocate resources more efficiently and comply with legal or regulatory requirements (Koller & O’Connor, 2005).
However, properly measuring harm inflicted by NWB has proven to be a quest in past years. Some scholars suggested that NWB could be characterized by the degree of harm it causes. From the target perspective, severity could be measured by the sum of the number of negative actions (Lutgen- Sandvik, Tracy and Alberts,2007). This implies that as the degree of NWB increases, so do the negative outcomes. Other scholars emphasized that it is not the objective behavior but how victims perceive and evaluate this behavior that determines the degree of harm (Frese & Zapf, 1988; Lazarus & Folkeman, 1986). Therefore, Agervold (2007) advocated to identify certain types of behaviors, independent of the target perspective.
One application of the above insights is that we inventory harm with a separate Part II, independent of the NWB types identified. This part does not ask the harm from the target perspective. We ask the respondent to provide an observation of the impact on their environment. This also allows us to inventory the harm for organizations and society that many studies on NWB underline such as sick leave, damaged reputation, premature retirement.

3.3.8. INWBQ

In Table 6, we present the INWBQ Part I with 69 items on NWBs from existing scales, with in-depth multiple-choice questions on 5 actor roles, and 7 items on NWB patterns. Harm is assessed with 11 items (including 50 multiple-choice) in Part II of this questionnaire. For the transparency of this development study, we indicate in Table 6 INWBQ the source of each item and the ranking number of this NWB or harm in the subcategory of its dimension. (See Table 6 from Supplementary Materials).

4. Discussion

The purpose of this study was to develop a questionnaire that integrates the content of measurement instruments of 16 most cited NWB constructs such as bullying, aggression, discrimination. This was needed since the plurality of different conceptualizations and operationalizations of NWB overlap, threatening their discriminant validity of measurement (Shaffer et al., 2016). In addition, research showed 83.5% of different types of NWBs occur simultaneous and sequentially (Privitera and Campbell, 2009), and involve multiple internal and external actor types (Merchant & Lundell, 2001), with more roles than perpetrator and target (Paull et al., 2012). Finally, measuring separate NWBs from one source limits our insights on the specific nature of its inflicted harm to different actor types and roles. It became highly necessary to develop a measurement tool integrating these factors. Our result is the Integrative NWB Questionnaire (INWBQ) which covers five dimensions of NWB in four occurrence patterns, four harms, four actor types, and six actor roles.

4.1. Scientific implications

Since the vast majority of current NWB tools focus on physical, psychological, and material NWB, the major gain of this tool is the addition of the sociocultural and digital dimensions. NWB based on gender or race as measured by the EHE and the SEQ is not a substantially different NWB, it only differs on a sociocultural ground. In this dimension, other sociocultural elements such as religion, ideology could also be added which make specific measurement tools unnecessary. The digital dimension contains the scientific implication that NWB can take place enabled and dependent by a digital medium. Instruments such as the ICA-W measure traditional NWB enabled by a digital medium such as mail or GSM. The addition of the cyber dependent element in the INWBQ allows us to get a picture of hybrid NWB forms in which traditional NWB co-occurs with NWB such as identity theft.
Another important implication concerns our broad inventory of four actor types, six actor roles and their harm. Scholars have been urging for several years that data should not only be collected from employees in the target role (Neall & Tuckey, 2014). The majority collect data from this source. The INWBQ also identifies external actors in modern workplaces as part of NWB, such as the public, clients, students, and their relatives. These actors are questioned on their mixt roles of target, perpetrator, assistant, defender, outsider, and witness. This will yield new scientific insights on actor types and their roles in NWB.
Finally, asking bodily harm, material damage, mental and social harm on the above mentioned four actor types and six roles is another important scientific implication. It is known that bystanders roles in the organization develop complaints (Lutgen-Sandvik, 2006; Mayhew et al., 2004). How this varies by role and externality provides new knowledge.

4.2. Practical implication

A Practical implication point is our questionnaire has 76 items in Part I and 11 (50 Multiple choice) items in Part II, causing the risk of non-response. Therefore, attention to response increasing procedures is necessary. In general, web survey design requires extra attention to procedures to manage the risk of non-response, coverage, and sampling errors (Vicente & Reis, 2010). On the other hand, this Internet context provides additional opportunities to make the survey attractive, such as providing context and incremental sampling to reduce dropouts (Couper et al., 2001). By grouping the harm-items we reduced the time taken to complete the survey influencing the response rate (Fuchs et al., 2000). Furthermore, we suggest stepwise administration of the questionnaire by sending it in parts to the respondent over a period.
Another practical implication is that taking separate modules from Part I of the INWBQ is not recommended to get a complete picture of an organizational NWB problem because the probability of 83,5% is much higher that combinations of NWB will occur (Privitera and Campbell, 2009). On the other hand, short scales as the GHS (Garthus-Niegel et al., 2016) and the LWMS (Steffgen et al., 2019) were developed for the for usage in different research contexts. The first scale for the Copenhagen Psychosocial Questionnaire (COPSOQ) a national standard workplace survey, the second scale for psychological research. For psychological research, the dimensions of the INWBQ can be used as separate modules. For national surveys, these separate dimensions are less suitable and further research could develop an abbreviated INWBQ considering the individual types of NWB.
Likewise, we do not recommend using individual modules of Part II of the INWBQ. Porath and Pearson (2010) did an excellent job of describing how one type of harm in time causes other types. Various health and mental complaints turn into persistent complaints that negatively affect the production process and the reduced reputation of the organization. It also causes material damage through, for example, increased staff turnover and intervention costs.
It is of course possible to take Part II separately from Part I when only a picture of the harm and damage is needed and the NWB is known. This can provide additional information for other methods that map harm, such as sick leave analyses, customer complaints or turnover rates.

4.3. Limitations

A limitation of this study is that we could not build on measuring some types of harm from existing NWB instruments[3]. We have added these from additional studies on harm of NWB. As a consequence, we assume but do not know whether for instance sleep difficulties caused by bullying (Hansen et al., 2014) are also caused by other types of NBW. By integrally including harm in the measurement of various types of NWB, this may provide new ways for future research.

4.4. Future research

We excluded coping because this behavior was not within our baseline definition. Nevertheless, in the NWB instruments we found several items of 7 coping styles: active problem solving (8), social support seeking (8), avoidance behavior (8), palliative reaction (7) depressive reaction (11), expression of emotions (10), and comforting cognitions (1) (coping styles after Schreurs et al., 1993). Recent research suggests that these styles may mitigate harm, but do not include an effective style for de-escalation of NWB (Hallett & Dickens, 2015). Therefore, for the inclusion of coping styles within NWB instruments, we believe future research could focus on inclusion of de-escalating styles such as appreciating, reducing anxiety, providing guidance, maintaining appropriate distance from a person (EDABS, (Mavandadi et al., 2016).
Despite these limitations, we believe we have taken a substantial step forward with the INWBQ to give a complete picture of the facets involved in NWB. This brings out much more and valid information for organizational consultants, psychologists, and scientists to work with. The analysis NWBs across several roles brings new insights to implement specific bystander interventions. For instance, to increase positive behaviors as defending a target while simultaneously decreasing NWB as assisting a perpetrator. The analysis of four types of inflicted harm brings new insights on total costs and needed care. This offers opportunities for system-based interventions including multiple stakeholders.

5. Conclusion

To our knowledge, this is the first attempt to develop an integrated measurement instrument from the various measurement instruments of NWB constructs. We succeeded in shaping the content of the INWBQ from 44 NWB construct instruments and 48 additional studies. With this method we wanted to provide optimal content validity. In this paper we gave a transparent description of the ways we tapped data, built subcategories, selected items, and made adjustments.
Although further analytical study on refinement and validation is needed this instrument fills an important research gap. Namely, it allows examination of four actor types in six actor roles on five types of NWB. In addition, the tool provides a broad scope on the inflicted bodily harm, material damage, mental and social harm of these actor types and roles.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

Acknowledgments

We are very grateful for the contribution the three experts made to our study. They set us to work with outstanding insights and additional sources.

References

  1. Adams, J. S. (1965). Inequity In Social Exchange. In Advances in Experimental Social Psychology (Vol. 2, Number C, pp. 267–299). [CrossRef]
  2. Alhabash, S.; McAlister, A.R.; Hagerstrom, A.; Quilliam, E.T.; Rifon, N.J.; Richards, J.I. Between Likes and Shares: Effects of Emotional Appeal and Virality on the Persuasiveness of Anticyberbullying Messages on Facebook. Cyberpsychology, Behav. Soc. Netw. 2013, 16, 175–182. [Google Scholar] [CrossRef] [PubMed]
  3. Andersson, L.M.; Pearson, C.M. Tit for Tat? The Spiraling Effect of Incivility in the Workplace. Acad. Manag. Rev. 1999, 24, 452. [Google Scholar] [CrossRef]
  4. Aquino, K.; Lamertz, K. A Relational Model of Workplace Victimization: Social Roles and Patterns of Victimization in Dyadic Relationships. J. Appl. Psychol. 2004, 89, 1023–1034. [Google Scholar] [CrossRef]
  5. Arenas, A.; Giorgi, G.; Montani, F.; Mancuso, S.; Perez, J.F.; Mucci, N.; Arcangeli, G. Workplace Bullying in a Sample of Italian and Spanish Employees and Its Relationship with Job Satisfaction, and Psychological Well-Being. Front. Psychol. 2015, 6, 1912. [Google Scholar] [CrossRef]
  6. Arvey, R.D.; Cavanaugh, M.A. Using Surveys to Assess the Prevalence of Sexual Harassment: Some Methodological Problems. J. Soc. Issues 1995, 51, 39–52. [Google Scholar] [CrossRef]
  7. Tatar, Z.B.; Yuksel, S. Mobbing at Workplace - Psychological Trauma and Documentation of Psychiatric Symptoms. Noro Psikiyatr. Arsivi 2018, 56, 57–62. [Google Scholar] [CrossRef]
  8. Barboza, G.E.; Schiamberg, L.B.; Oehmke, J.; Korzeniewski, S.J.; Post, L.A.; Heraux, C.G. Individual Characteristics and the Multiple Contexts of Adolescent Bullying: An Ecological Perspective. J. Youth Adolesc. 2009, 38, 101–121. [Google Scholar] [CrossRef]
  9. Baron, R. A., & Neuman, J. H. (1996). Workplace violence and workplace aggression: Evidence on their relative frequency and potential causes. Aggressive Behavior, 22(3), 161–173. [CrossRef]
  10. Bayramoğlu, M.M.; Toksoy, D. Leadership and Bullying in the Forestry Organization of Turkey. BioMed Res. Int. 2017, 2017, 1–11. [Google Scholar] [CrossRef]
  11. Beck, A.T.; Steer, R.A.; Brown, G. Beck Depression Inventory–II; Psychological Corporation: San Antonio, TX, USA, 1996. [Google Scholar] [CrossRef]
  12. Bennett, R.J.; Robinson, S.L. Development of a measure of workplace deviance. J. Appl. Psychol. 2000, 85, 349–360. [Google Scholar] [CrossRef] [PubMed]
  13. Björkqvist, K.; Österman, K.; Lagerspetz, K.M.J. Sex differences in covert aggression among adults. Aggress. Behav. 1994, 20, 27–33. [Google Scholar] [CrossRef]
  14. Black, S. M., Creese, S., Guest, R. M., Pike, B., Saxby, S. J., Fraser, D. S., Stevenage, S. V, & Whitty, M. T. (2012). Superidentity: Fusion of identity across real and cyber domains. Global Forum on Identity, ID 360, 1–10. https://eprints.soton.ac.uk/336645/1/ID360_-_finalpaper.pdf.
  15. Bougie, R., & Sekaran, U. (2020). Research methods for business: A skill building approach (8th ed.). John Wiley & Sons. Inc. https://www.wiley.com/en-us/Research+Methods+For+Business%3A+A+Skill+Building+Approach%2C+8th+Edition-p-9781119561248.
  16. Bowling, N.A.; Beehr, T.A. Workplace harassment from the victim's perspective: A theoretical model and meta-analysis. J. Appl. Psychol. 2006, 91, 998–1012. [Google Scholar] [CrossRef] [PubMed]
  17. Broersen, A., Ossenblok, A., & Montesano Montessori, N. (2015). Pesten en sociale veiligheid op scholen: definities en keuzes op micro-, meso- en macroniveau [Bullying and social safety in schools: definitions and choices at micro, meso and macro level]. Tijdschrift Voor Orthopedagogiek, 54(54), 108–119. http://schoolpsychologencongres.nl/images/presentaties/Bijbehorend-artikel-wokshop-Annemiek-Broersen.pdf.
  18. Campo, V.R.; Klijn, T.P. Verbal abuse and mobbing in pre-hospital care services in Chile. Rev. Latino-Americana de Enferm. 2018, 25, e2956. [Google Scholar] [CrossRef]
  19. Cannavò, M.; La Torre, F.; Sestili, C.; La Torre, G.; Fioravanti, M. Work Related Violence As A Predictor Of Stress And Correlated Disorders In Emergency Department Healthcare Professionals. Clin. Ther. 2019, 170, e110–e123. [Google Scholar] [CrossRef]
  20. Carrera, M.V.; DePalma, R.; Lameiras, M. Toward a More Comprehensive Understanding of Bullying in School Settings. Educ. Psychol. Rev. 2011, 23, 479–499. [Google Scholar] [CrossRef]
  21. Cele, B. (2018). Addendum to the saps annual report. South African Police Service. https://nationalgovernment.co.za/department_annual/296/2019-south-african-police-service-(saps)-annual-report.pdf.
  22. Cheung, T.; Lee, P.H.; Yip, P.S.F. The association between workplace violence and physicians’ and nurses’ job satisfaction in Macau. PLOS ONE 2018, 13, e0207577. [Google Scholar] [CrossRef] [PubMed]
  23. Codina, M.; Pereda, N.; Guilera, G. Lifetime Victimization and Poly-Victimization in a Sample of Adults With Intellectual Disabilities. J. Interpers. Violence 2020, 37, 2062–2082. [Google Scholar] [CrossRef] [PubMed]
  24. Colquitt, J. A., & Rodell, J. B. (2015). Measuring justice and fairness. In R. S. Cropanzano & M. L. Ambrose (Eds.), The Oxford Handbook of Justice in the Workplace (Number March 2019, pp. 1–30). Oxford University Press. [CrossRef]
  25. Couper, M.P.; Traugott, M.W.; Lamias, M.J. Web Survey Design and Administration. Public Opin. Q. 2001, 65, 230–253. [Google Scholar] [CrossRef] [PubMed]
  26. Cropanzano, R.; Bowen, D.E.; Gilliland, S.W. The Management of Organizational Justice. Acad. Manag. Perspect. 2007, 21, 34–48. [Google Scholar] [CrossRef]
  27. Dalal, R.S. A Meta-Analysis of the Relationship Between Organizational Citizenship Behavior and Counterproductive Work Behavior. J. Appl. Psychol. 2005, 90, 1241–1255. [Google Scholar] [CrossRef] [PubMed]
  28. Demaray, M.K.; Summers, K.H.; Jenkins, L.N.; Becker, L.D. Bullying Participant Behaviors Questionnaire (BPBQ): Establishing a Reliable and Valid Measure. J. Sch. Violence 2016, 15, 158–188. [Google Scholar] [CrossRef]
  29. DeSouza, E.R. Frequency Rates and Correlates of Contrapower Harassment in Higher Education. J. Interpers. Violence 2011, 26, 158–188. [Google Scholar] [CrossRef]
  30. Einarsen, S. Harassment and bullying at work: A review of the scandinavian approach. Aggress. Violent Behav. 2000, 5, 379–401. [Google Scholar] [CrossRef]
  31. Einarsen, S.; Hoel, H.; Notelaers, G. Measuring exposure to bullying and harassment at work: Validity, factor structure and psychometric properties of the Negative Acts Questionnaire-Revised. Work. Stress 2009, 23, 24–44. [Google Scholar] [CrossRef]
  32. Einarsen, S, Hoel, H., Zapf, D., & Cooper, C. L. (2011). Bullying and harassment in the workplace: Developments in theory, research, and practice (Second). CRC Press. ISBN 9781439804896 - CAT# K10270.
  33. Einarsen, S.; Skogstad, A. Bullying at work: Epidemiological findings in public and private organizations. Eur. J. Work. Organ. Psychol. 1996, 5, 185–201. [Google Scholar] [CrossRef]
  34. Einarsen, Ståle V., Hoel, H., Zapf, D., & Cooper, C. L. (2020). The concept of bullying and harassment at work: The european tradition. In S. V. Einarsen, H. Hoel, D. Zapf, & C. L. Cooper (Eds.), Bullying and harassment in the workplace: Theory, Research and Practice (pp. 3–41). CRC Press.
  35. Elovainio, M.; Kivimäki, M.; Vahtera, J. Organizational Justice: Evidence of a New Psychosocial Predictor of Health. Am. J. Public Heal. 2002, 92, 105–108. [Google Scholar] [CrossRef] [PubMed]
  36. Escartín, J.; Monzani, L.; Leong, F.; Rodríguez-Carballeira. A reduced form of the Workplace Bullying Scale – the EAPA-T-R: A useful instrument for daily diary and experience sampling studies. Work. Stress 2017, 31, 42–62. [Google Scholar] [CrossRef]
  37. Escartín, J.; Rodríguez-Carballeira, A.; Zapf, D.; Porrúa, C.; Martín-Peña, J. Perceived severity of various bullying behaviours at work and the relevance of exposure to bullying. Work. Stress 2009, 23, 191–205. [Google Scholar] [CrossRef]
  38. Escartín, J., Vranjes, I., Baillien, E., & Notelaers, G. (2021). Workplace Bullying and Cyberbullying Scales: An Overview. In R. C. D’Cruz P., Noronha E., Notelaers G. (Ed.), Concepts, Approaches and Methods. Handbooks of Workplace Bullying, Emotional Abuse and Harassment (1st ed., pp. 325–368). Springer, Singapore.
  39. Every-Palmer, S.; Barry-Walsh, J.; Pathé, M. Harassment, stalking, threats and attacks targeting New Zealand politicians: A mental health issue. Aust. New Zealand J. Psychiatry 2015, 49, 634–641. [Google Scholar] [CrossRef]
  40. Farley, S.; Coyne, I.; Axtell, C.; Sprigg, C. Design, development and validation of a workplace cyberbullying measure, the WCM. Work. Stress 2016, 30, 293–317. [Google Scholar] [CrossRef]
  41. Ferrer, B.M.; Ruiz, D.M.; Amador, L.V.; Orford, J. School Victimization Among Adolescents. An Analysis from an Ecological Perspective. Psychosoc. Interv. 2011, 20, 149–160. [Google Scholar] [CrossRef]
  42. Fox, S.; Spector, P.E.; Miles, D. Counterproductive Work Behavior (CWB) in Response to Job Stressors and Organizational Justice: Some Mediator and Moderator Tests for Autonomy and Emotions. J. Vocat. Behav. 2001, 59, 291–309. [Google Scholar] [CrossRef]
  43. Fredericksen, E.D.; McCorkle, S. Explaining Organizational Responses to Workplace Aggression. Public Pers. Manag. 2013, 42, 223–238. [Google Scholar] [CrossRef]
  44. Fuchs, M., Couper, M. P., & Hansen, S. E. (2000). Technology effects: Do capi or Papi interviews take longer? Journal of Official Statistics, 16(3), 273–286. https://www.scb.se/contentassets/f6bcee6f397c4fd68db6452fc9643e68/technology-effects-do-capi-or-papi-interviews-take-longer.pdf.
  45. Furnell, S. (2002). Cybercrime: Vandalizing the information society. Addison-Wesley.
  46. Garthus-Niegel, S.; Nübling, M.; Letzel, S.; Hegewald, J.; Wagner, M.; Wild, P.S.; Blettner, M.; Zwiener, I.; Latza, U.; Jankowiak, S.; et al. Development of a mobbing short scale in the Gutenberg Health Study. Int. Arch. Occup. Environ. Heal. 2016, 89, 137–146. [Google Scholar] [CrossRef]
  47. Giga, S. I., Hoel, H., & Lewis, D. (2008). The costs of workplace bullying. https://www.researchgate.net/publication/260246863_The_Costs_of_Workplace_Bullying.
  48. Glick, P. (2005). Choice of Scapegoats. In J. F. Dovidio, P. Glick, & L. Rudman (Eds.), On the Nature of Prejudice: Fifty Years after Allport. (pp. 244–261). Blackwell Publishing. [CrossRef]
  49. Gofin, R.; Avitzour, M. Traditional Versus Internet Bullying in Junior High School Students. Matern. Child Heal. J. 2012, 16, 1625–1635. [Google Scholar] [CrossRef]
  50. Gruys, M.L.; Sackett, P.R. Investigating the Dimensionality of Counterproductive Work Behavior. Int. J. Sel. Assess. 2003, 11, 30–42. [Google Scholar] [CrossRef]
  51. Gurrola-Pena, G.M.; Balcazar-Nava, P.E.; del Villar, O.E.; Lozano-Razo, G.; Zavala-Rayas, J. Construction and Validation of the Contextual Victimization Questionnaire (CVCV) with Mexican Young Adults. Int. J. Emerg. Ment. Heal. Hum. Resil. 2018, 20, 1–7. [Google Scholar] [CrossRef]
  52. Hallett, N.; Dickens, G.L. De-escalation: A survey of clinical staff in a secure mental health inpatient service. Int. J. Ment. Heal. Nurs. 2015, 24, 324–333. [Google Scholar] [CrossRef]
  53. Hansen, .M.; Hogh, A.; Garde, A.H.; Persson, R. Workplace bullying and sleep difficulties: a 2-year follow-up study. Int. Arch. Occup. Environ. Heal. 2014, 87, 285–294. [CrossRef]
  54. Haynie, D.L.; Nansel, T.; Eitel, P.; Crump, A.D.; Saylor, K.; Yu, K.; Simons-Morton, B. Bullies, Victims, and Bully/Victims:. J. Early Adolesc. 2001, 21, 29–49. [Google Scholar] [CrossRef]
  55. Heerde, J.A.; Hemphill, S.A. Are Bullying Perpetration and Victimization Associated with Adolescent Deliberate Self-Harm? A Meta-Analysis. Arch. Suicide Res. 2019, 23, 353–381. [Google Scholar] [CrossRef]
  56. Henrichsen, J. R., Betz, M., & Lisosky, J. M. (2015). Building Digital Safety for Journalism: a Survey of Selected Issues. In Unesco Series on Internet Freedom. http://unesdoc.unesco.org/images/0023/002323/232358e.pdf.
  57. Hitlan, R. T., Cliffton, R. J., & DeSoto, C. M. (2006). Perceived exclusion in the workplace: The moderating effects of gender on workrelated attitudes and psychological health. North American Journal of Psychology, 8(2), 217–236.
  58. Hogh, A.; Hansen, M.; Mikkelsen, E.G.; Persson, R. Exposure to negative acts at work, psychological stress reactions and physiological stress response. J. Psychosom. Res. 2012, 73, 47–52. [Google Scholar] [CrossRef] [PubMed]
  59. Holroyd-Leduc, J.M.; Straus, S.E. #MeToo and the medical profession. Can. Med Assoc. J. 2018, 190, E972–E973. [Google Scholar] [CrossRef]
  60. Hoobler, J.M.; Brass, D.J. Abusive supervision and family undermining as displaced aggression. J. Appl. Psychol. 2006, 91, 1125–1133. [Google Scholar] [CrossRef]
  61. Hyde, M.; Jappinen, P.; Theorell, T.; Oxenstierna, G. Workplace conflict resolution and the health of employees in the Swedish and Finnish units of an industrial company. Soc. Sci. Med. 2006, 63, 2218–2227. [Google Scholar] [CrossRef]
  62. Ireland, J. (2013). Bullying among prisoners: Innovations in theory and research. In J. L. Ireland (Ed.), Bullying among prisoners: The need for innovation. (2nd ed., pp. 1–30). Willan. [CrossRef]
  63. Jacob, L.; Kostev, K. Conflicts at work are associated with a higher risk of cardiovascular disease. 2017, 15, 1–8. [CrossRef]
  64. Juvonen, J., & Graham, S. (2016). Peer harassment in school (J. Juvonen & S. Graham (eds.); 1st ed.). The Guilford Press.
  65. Kacmar, K. M., & Carlson, D. S. (1997). Further validation of the perceptions of politics scale (POPS): A multiple sample investigation. Journal of Management, 23(5), 627–658. [CrossRef]
  66. Kaukiainen, A.; Salmivalli, C.; Björkqvist, K.; Österman, K.; Lahtinen, A.; Kostamo, A.; Lagerspetz, K. Overt and covert aggression in work settings in relation to the subjective well-being of employees. Aggress. Behav. 2001, 27, 360–371. [Google Scholar] [CrossRef]
  67. Keashly, L. Interpersonal and Systemic Aspects of Emotional Abuse at Work: The Target’s Perspective. Violence Vict. 2001, 16, 233–268. [Google Scholar] [CrossRef]
  68. Khan, A.K.; Quratulain, S.; Bell, C.M. Episodic envy and counterproductive work behaviors: Is more justice always good? J. Organ. Behav. 2014, 35, 128–144. [Google Scholar] [CrossRef]
  69. Khattak, M.N.; Zolin, R.; Muhammad, N. The combined effect of perceived organizational injustice and perceived politics on deviant behaviors. Int. J. Confl. Manag. 2021, 32, 62–87. [Google Scholar] [CrossRef]
  70. Kivimaki, M.; Virtanen, M.; Vartia, M.; Elovainio, M.; Vahtera, J.; Keltikangas-Järvinen, L. Workplace bullying and the risk of cardiovascular disease and depression. Occup. Environ. Med. 2003, 60, 779–783. [Google Scholar] [CrossRef]
  71. Koller, G., & O’Connor, M. K. (2005). Risk Assessment and Decision Making in Business and Industry: A Practical Guid (2nd ed.). Chapman and Hall/CRC. https://www.routledge.com/Risk-Assessment-and-Decision-Making-in-Business-and-Industry-A-Practical/Koller/p/book/9780367393151.
  72. Laumann, E., Marsden, P. V., & Prenski, D. (1983). The boundary specification problem in network analysis (applied for conference Methods in Social Network Analysis) (R. Burt & M. Minor (eds.); 1st ed.). Sage Publications, Inc. https://www.researchgate.net/publication/238338190.
  73. Levine, M.; Taylor, P.J.; Best, R. Third Parties, Violence, and Conflict Resolution. Psychol. Sci. 2011, 22, 406–412. [Google Scholar] [CrossRef]
  74. Leymann, H. Mobbing and Psychological Terror at Workplaces. Violence Vict. 1990, 5, 119–126. [Google Scholar] [CrossRef]
  75. Leymann, H. The content and development of mobbing at work. Eur. J. Work. Organ. Psychol. 1996, 5, 165–184. [Google Scholar] [CrossRef]
  76. Lutgen-Sandvik, P. Take This Job and … : Quitting and Other Forms of Resistance to Workplace Bullying. Commun. Monogr. 2006, 73, 406–433. [Google Scholar] [CrossRef]
  77. Marcus, B.; Schuler, H.; Quell, P.; Hümpfner, G. Measuring Counterproductivity: Development and Initial Validation of a German Self-Report Questionnaire. Int. J. Sel. Assess. 2002, 10, 18–35. [Google Scholar] [CrossRef]
  78. Mathisen, G.E.; Einarsen, S.; Mykletun, R. The occurrences and correlates of bullying and harassment in the restaurant sector. Scand. J. Psychol. 2008, 49, 59–68. [Google Scholar]
  79. Mavandadi, V.; Bieling, P.J.; Madsen, V. Effective ingredients of verbal de-escalation: validating an English modified version of the ‘De-Escalating Aggressive Behaviour Scale’. J. Psychiatr. Ment. Heal. Nurs. 2016, 23, 357–368. [Google Scholar] [CrossRef]
  80. Mayhew, C.; McCarthy, P.; Chappell, D.; Quinlan, M.; Barker, M.; Sheehan, M. Measuring the Extent of Impact from Occupational Violence and Bullying on Traumatised Workers. Empl. Responsib. Rights J. 2004, 16, 117–134. [Google Scholar] [CrossRef]
  81. A Merchant, J.; A Lundell, J. Workplace Violence Intervention Research Workshop, April 5–7, 2000, Washington, DC: Background, rationale, and summary. Am. J. Prev. Med. 2001, 20, 135–140. [Google Scholar] [CrossRef]
  82. Nagle, J. Twitter, cyber-violence, and the need for a critical social media literacy in teacher education: A review of the literature. Teach. Teach. Educ. 2018, 76, 86–94. [Google Scholar] [CrossRef]
  83. Namie, G., & Namie, R. (2000). Workplace bullying: The silent epidemic. Employee Rights Quarterly, 1(2), 1–12.
  84. Neall, A.M.; Tuckey, M.R. A methodological review of research on the antecedents and consequences of workplace harassment. J. Occup. Organ. Psychol. 2014, 87, 225–257. [Google Scholar] [CrossRef]
  85. Neuman, J.H.; Baron, R.A. Workplace Violence and Workplace Aggression: Evidence Concerning Specific Forms, Potential Causes, and Preferred Targets. J. Manag. 1998, 24, 391–419. [Google Scholar] [CrossRef]
  86. Nielsen, M.B.; Emberland, J.S.; Knardahl, S. Workplace Bullying as a Predictor of Disability Retirement. J. Occup. Environ. Med. 2017, 59, 609–614. [Google Scholar] [CrossRef]
  87. Nielsen, M.B.; Knardahl, S. Is workplace bullying related to the personality traits of victims? A two-year prospective study. Work. Stress 2015, 29, 128–149. [Google Scholar] [CrossRef]
  88. NIOSH. (2023). Occupational violence. Acessed January 10. https://www.cdc.gov/niosh/topics/violence/default.html.
  89. Notelaers, G.; Van der Heijden, B.; Hoel, H.; Einarsen, S. Measuring bullying at work with the short-negative acts questionnaire: identification of targets and criterion validity. Work. Stress 2019, 33, 58–75. [Google Scholar] [CrossRef]
  90. O'Connor, M.; Gutek, B.A.; Stockdale, M.; Geer, T.M.; Melançon, R. Explaining sexual harassment judgments: looking beyond gender of the rater. Law Hum. Behav. 2004, 28, 69–95. [Google Scholar] [CrossRef]
  91. Paull, M.; Omari, M.; Standen, P. When is a bystander not a bystander? A typology of the roles of bystanders in workplace bullying. Asia Pac. J. Hum. Resour. 2012, 50, 351–366. [Google Scholar] [CrossRef]
  92. Poon, J.M. Situational antecedents and outcomes of organizational politics perceptions. J. Manag. Psychol. 2003, 18, 138–155. [Google Scholar] [CrossRef]
  93. Porath, C.L.; Pearson, C.M. The Cost of Bad Behavior. Organ. Dyn. 2010, 39, 64–71. [Google Scholar] [CrossRef]
  94. Porath, C.L.; Pearson, C.M. Emotional and Behavioral Responses to Workplace Incivility and the Impact of Hierarchical Status. J. Appl. Soc. Psychol. 2012, 42, E326–E357. [Google Scholar] [CrossRef]
  95. Privitera, C.; Campbell, M.A.; Mpsych; Ph.D.; Forssell, R.; Coyne, I.; Farley, S.; Axtell, C.; Sprigg, C.; Best, L.; et al. Cyberbullying: The New Face of Workplace Bullying?. CyberPsychology Behav. 2009, 12, 395–400. [CrossRef]
  96. Quigley, E., Ruggiero, J., McEvoy, T., Hollister, S., Schwoebel, A., & Inacker, P. (2020). Addressing workplace violence and aggression in health care settings: One unit’s journey. Nurse Leader, in press.
  97. A Richman, J.; Rospenda, K.M.; Nawyn, S.J.; A Flaherty, J.; Fendrich, M.; Drum, M.L.; Johnson, T.P. Sexual harassment and generalized workplace abuse among university employees: prevalence and mental health correlates. Am. J. Public Heal. 1999, 89, 358–363. [Google Scholar] [CrossRef]
  98. Rivera, J., & Abuín, M. (2003). Cuestionario de estrategias de acoso en el trabajo: El LIPT-60 (Leymann Inventory of Pychological Terrorization) en version espa- ñola [Questionnaire of etrategies of harrassment at the workplace: El LIPT-60 (Leymann Inventory of Pychological Terrorizatio. Psiquis, 24(2), 59–69. https://www.academia.edu/33980625/Cuestionario_de_estrategias_de_acoso_psicologico_Leymann_Inventory_of_Psychological_Terrorization.
  99. Robinson, M.A. Using multi-item psychometric scales for research and practice in human resource management. Hum. Resour. Manag. 2018, 57, 739–750. [Google Scholar] [CrossRef]
  100. Robinson, S.L.; Bennett, R.J. A Typology of Deviant Workplace Behaviors: A Multidimensional Scaling Study. Acad. Manag. J. 1995, 38, 555–572. [Google Scholar] [CrossRef]
  101. Rospenda, K.M.; Richman, J.A.; Shannon, C.A. Prevalence and Mental Health Correlates of Harassment and Discrimination in the Workplace. J. Interpers. Violence 2009, 24, 819–843. [Google Scholar] [CrossRef]
  102. Rosta, J.; Aasland, O.G. Perceived bullying among Norwegian doctors in 1993, 2004 and 2014–2015: a study based on cross-sectional and repeated surveys. BMJ Open 2018, 8, e018161. [Google Scholar] [CrossRef]
  103. Ruiz-Hernández, J.A.; López-García, C.; Llor-Esteban, B.; Galián-Muñoz, I.; Benavente-Reche, A.P. Evaluation of the users violence in primary health care: Adaptation of an instrument. Int. J. Clin. Heal. Psychol. 2016, 16, 295–305. [Google Scholar] [CrossRef]
  104. Sackett, P., & DeVore, C. (2001). Counterproductive behaviours at work. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial work and organizational psychology. Personal Psychology (1st ed., pp. 145–164). Sage Publications, Inc.
  105. Scholte, R., Nelen, W., De Wit, W., & Kroes, G. (2016). Sociale veiligheid in en rond scholen [Social safety in and around schools] 2010-2016 (Number december). https://www.rijksoverheid.nl/documenten/rapporten/2016/12/23/sociale-veiligheid-in-en-rond-scholen.
  106. Schreurs, P. J. G., G. van de Willige, J., Brosschot, .F., Graus, G. M. H., & Tellegen, B. (1993). De Utrechtse Coping Lijst (UCL): omgaan met problemen en gebeurtenissen (The Utrecht Coping List : coping with problems and events) (revised ed). Pearson.
  107. Shaffer, J.A.; DeGeest, D.; Li, A. Tackling the Problem of Construct Proliferation. Organ. Res. Methods 2016, 19, 80–110. [Google Scholar] [CrossRef]
  108. Skarlicki, D.P.; Folger, R. Retaliation in the workplace: The roles of distributive, procedural, and interactional justice. J. Appl. Psychol. 1997, 82, 434–443. [Google Scholar] [CrossRef]
  109. Sliter, K.A.; Sliter, M.T.; Withrow, S.A.; Jex, S.M. Employee adiposity and incivility: Establishing a link and identifying demographic moderators and negative consequences. J. Occup. Heal. Psychol. 2012, 17, 409–424. [Google Scholar] [CrossRef]
  110. Spector, P. E., & Fox, S. (2005). The stressor-emotion model of Counterproductive Work Behavior. In S. Fox & P. E. Spector (Eds.), Counterproductive Work Behavior: Investigations of actors and targets. (pp. 151–174). American Psychological Association. [CrossRef]
  111. Spector, P. E., & Jex, S. M. (1998). Development of four self-report measures of job stressors and strain: Interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. Journal of Occupational Health Psychology, 3(4), 356–367. https://psycnet.apa.org/buy/1998-12418-005.
  112. Sprigg, C.A.; Niven, K.; Dawson, J.; Farley, S.; Armitage, C.J. Witnessing workplace bullying and employee well-being: A two-wave field study. J. Occup. Heal. Psychol. 2019, 24, 286–296. [Google Scholar] [CrossRef] [PubMed]
  113. Steffgen, G.; Sischka, P.; Schmidt, A.F.; Kohl, D.; Happ, C. The Luxembourg Workplace Mobbing Scale. Eur. J. Psychol. Assess. 2019, 35, 164–171. [Google Scholar] [CrossRef]
  114. Tabor, J.; Griep, Y.; Collins, R.; Mychasiuk, R. Investigating the Neurological Correlates of Workplace Deviance Using a Rodent Model of Extinction. Sci. Rep. 2018, 8, 1–8. [Google Scholar] [CrossRef]
  115. Takaki, J.; Taniguchi, T.; Hirokawa, K. Associations of Workplace Bullying and Harassment with Pain. Int. J. Environ. Res. Public Heal. 2013, 10, 4560–4570. [Google Scholar] [CrossRef]
  116. Thomas, S.P.; Burk, R. Junior nursing students' experiences of vertical violence during clinical rotations. Nurs. Outlook 2009, 57, 226–231. [Google Scholar] [CrossRef]
  117. Thompson, M.J.; Carlson, D.S.; Kacmar, K.M.; Vogel, R.M. The cost of being ignored: Emotional exhaustion in the work and family domains. J. Appl. Psychol. 2020, 105, 186–195. [Google Scholar] [CrossRef]
  118. Thurston, R.C.; Chang, Y.; Matthews, K.A.; von Känel, R.; Koenen, K. Association of Sexual Harassment and Sexual Assault With Midlife Women’s Mental and Physical Health. JAMA Intern. Med. 2019, 179, 48–53. [Google Scholar] [CrossRef]
  119. Verkuil, B., Atasayi, S., & Molendijk, M. L. (2015). Workplace bullying and mental health : A meta- analysis on cross-sectional and longitudinal data. PLoS ONE, 10(8), 1–17. [CrossRef]
  120. Verschuren, C.M.; Tims, M.; de Lange, A.H. A Systematic Review of Negative Work Behavior: Toward an Integrated Definition. Front. Psychol. 2021, 12. [Google Scholar] [CrossRef]
  121. Vicente, P.; Reis, E. Using Questionnaire Design to Fight Nonresponse Bias in Web Surveys. Soc. Sci. Comput. Rev. 2010, 28, 251–267. [Google Scholar] [CrossRef]
  122. Weiss, D., Marmar, C., Wilson, J., & Keane, T. (1997). Assessing psychological trauma and PTSD (Guilford Press (ed.)).
  123. Zivnuska, S.L.; Carlson, D.S.; Carlson, J.R.; Harris, K.J.; Harris, R.B.; Valle, M. Information and communication technology incivility aggression in the workplace: Implications for work and family. Inf. Process. Manag. 2020, 57, 102222. [Google Scholar] [CrossRef]
Preprints 74603 g001
Figure 1. Development INWBQ.
Figure 1. Development INWBQ.
Preprints 74603 g001
Table 2. Dimensions INWBQ.
Table 2. Dimensions INWBQ.
Preprints 74603 i001
Table 5. Dimensions Actor Types & Roles, Subcategories.
Table 5. Dimensions Actor Types & Roles, Subcategories.
Preprints 74603 i002

Notes

1
Due to limited space in this article, subcategories and their reduction are included in a Supplementary Table Reduction Process NWBs /harms that can be requested from the author. We suffice here with 2 subcategories of the dimension Material NWB, and one subcategory of the dimension Mental harm.
2
Michelle Demaray, Psychology Department, Northern Illinois University, DeKalb, Illinois, USA, Jorge Escartin Solanelles, Department of Social Psychology, Universitat de Barcelona, Barcelona, Spain, Kara Ng, Alliance Manchester Business School, The University of Manchester, Manchester, UK.
3
33 Bodily harms, 18 Material damage, 107 Mental harms, 7 Social harms.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated