Prerpints.org logo
Preprint
Article

A Literature Review and Taxonomy of In-VR Questionnaire User Interfaces

Altmetrics

Downloads

58

Views

35

Comments

0

This version is not peer-reviewed

Theories and Applications of Human-Computer Interaction

Submitted:

03 June 2024

Posted:

04 June 2024

You are already at the latest version

Alerts
Abstract
Previous research demonstrates that the interruption of immersive experiences may lead to a bias in the results of questionnaires. Thus, the traditional way of presenting questionnaires, paper-based or web-based, may not be compatible with evaluating VR experiences. Recent research has shown the positive impact of embedding questionnaires contextually into the virtual environment. However, a comprehensive overview of the available VR questionnaire solutions is currently missing. Furthermore, no clear taxonomy exists for these different solutions in the literature. To address this, we present a literature review of VR questionnaire user interfaces (UI) following PRISMA guidelines. Our search returned 1.109 initial results, which were screened for eligibility, resulting in a corpus of 25 papers. This paper contributes to HCI and games research with a literature review of embedded questionnaires in VR, discussing the advantages and disadvantages and introducing a taxonomy of in-VR questionnaire UIs.
Keywords: 
Subject: Computer Science and Mathematics  -   Computer Science

1. Introduction

Virtual reality (VR) remained a specialized technology for much of the last century until—in the previous decade—lighter-weight, consumer-friendly head-mounted displays (HMDs) were introduced, and games were pulling early adopters and industries into this space [1,2,3,4,5,6,7,8]. Many researchers use VR to conduct user studies using either objective methods and sensory data (e.g., respiration sensor, brain activity measurement, blood pressure, eye tracking [9,10,11]), or subjective methods and questionnaires (e.g., PXI [12], IPQ [13], PENS [14]). However, the difficulties in applying objective methods lead to the increasing use of subjective methods, especially questionnaires, for the evaluation process. [15,16,17].
The traditional methods for completing questionnaires have been either paper-based forms or web-based user interfaces. Although these ways of presenting questionnaires have been used for a long time and are well-developed for user needs, the immersive features of VR may not be compatible with them. Recent studies show that when the subject of the questionnaire is related to a feature of experience that can be interrupted by exiting the environment (e.g., perception of presence), implementing the questionnaire inside the environment can lead to better consistency and reliability of the results [9,15,18] as well as an increase in motivation, engagement, and presence of participants [19,20].
These influences may be even more dominant for immersive VR experiences, as one of the main features is the high degree of immersion [21]. For example, imagine a situation where the research objective is to measure presence during a psychiatric therapy session in VR. The traditional methods would require participants to leave the virtual environment and remove the headset to complete the questionnaires. This procedure can lead to an interruption of the user’s perception of presence and biased answers. Considering these points, recent studies investigate the influence of embedding questionnaires as an element directly in the virtual environment. This element can be presented as a separate user interface (UI) or as an interactive object in VR [21]. Accordingly, it is crucial to understand alternative methods of presenting questionnaires for VR studies to overcome the barriers of traditional workflows.
Evaluating the impact of embedding questionnaires in a VR environment can improve the research communities’ understanding of its benefits and limitations. Furthermore, enhancing the implementation procedure and removing barriers limiting the use of embedded questionnaires may lead to increased uptake. Daily increasing tendency to use VR applications and metaverse daily [22], emphasis a need for more detailed studies on VR. On the other hand, the traditional way of using questionnaires in VR may lead to biased results. In this paper, we review recent articles that either use embedded questionnaires as a method for their studies or as a research topic to investigate and improve this type of questionnaire representation and to answer the following questions:
  • RQ1: How does the literature refer to or define in-VR questionnaires?
  • RQ2: What are the most commonly used UIs and interactions for in-VR questionnaires?
  • RQ3: What are future research and development opportunities for in-VR questionnaires?
We use the term “in-VR,” referring to questionnaires embedded and integrated into VR. The term “out-VR” refers to the traditional paper-based or web-based representation of questionnaires. While our literature review points to works exploring different aspects of in-VR questionnaires, a precise overview of how they can be presented in virtual environments and how to interact with them is lacking. Furthermore, the terminology used to refer to in-VR questionnaires is imprecise. We contribute to human-computer interaction (HCI) research in games with a taxonomy of in-VR questionnaires, discussing common ways to present and interact with them in VR. It is important to investigate the impact of various design choices on the user experience of virtual reality (VR) in education. This is because the use of VR technology in education is rapidly increasing. To achieve this, using questionnaires to gather user feedback will be necessary. This paper can be useful in future studies on VR educational content by providing a review of potential VR questionnaire terminology and designs.

2. Related Work

A taxonomy is a collection of dimensions, each composed of features [23]. Examining related work on the taxonomy of various attributes of virtual reality experiences can help researchers in the HCI community identify current research frontiers and future directions. A review by Kim et al. [24] on 65 articles, categorizes the affecting factors on the user experience (UX) in VR based on user characteristics, device settings, and interaction types. In addition to these factors, the evaluation method can also influence the results and their interpretation.
Motejlek and Alpay [25] did a taxonomy focused on the application of virtual and augmented reality in education and based on FURPS+ [26] method. Their study results in categorizing the VR experience based on user interactions in four classes: General Purpose Controller, User Tracking, Specialized Controller, and No Interaction. Most VR interactions can be performed with gestures. In a study by Bhowmick et al. [27], a taxonomy of gestures for object selection in VR was developed. This taxonomy can be useful for interacting with different UI elements. According to their taxonomy, gestures in VR can be categorized into two main types: Hand dominance and movement with multiple body parts. Multiple body part movement can be as simple as walking or combining multiple body part gestures (this may require additional sensors to track body parts). Hand dominance gestures can rely only on the dominant hand or be performed in combination with gestures from the non-dominant hand (developers may also not consider a dominant hand). Another taxonomy by Ruscella and Obeid [28] tries to categorize the attributes of VR applications that lead to a successful immersive experience. Accordingly, they categorize the influencing factors in: Interactivity, Story, Gamification, Dynamics, Co-participation, Embodiment, Immersive Technology, Meta control, and Didactic Capacity. To analyze these factors, collecting and analyzing feedback from participants in user research studies can be an essential step[29]. Most HCI studies in recent years have addressed empirical and artifact contributions [30,31].
If the research topic is to consider the measurement of values related to participants’ beliefs and feelings about particular topics, researchers may use self-report measures (i.e., questionnaires) [32]. A questionnaire may contain standardized and structured questions based on previous studies to elicit specific results or free-form questions to reflect user preferences without limitation of standard questions [33]. As with other research methods, the influencing factors that lead to bias in the results should be reduced in the design and implementation of the questionnaire. In a study by Choi et al. [34], 48 types of bias in self-report surveys are discussed and divided into three main types: Question Design, Questionnaire Design, and Questionnaire Administration. In the case of immersive experiences, Alexandrovsky et al. [15] argue, based on a study by Schwind et al. [35], that consistency in questionnaire administration can lead to a reduction in random errors. An example of these errors can be conditions that lead to a disruption of the sense of presence by switching between real and virtual environments.
A study by [36] shows that leaving a virtual environment can lead to disorientation. The extent of this disorientation depends on the duration of the VR experience, the degree of presence in the VR environment, and the distance traveled. They found that this disorientation even occurs in simplified virtual environments. These types of interruptions can negatively impact user engross involvement (a level of involvement in the virtual environment that emotionally affects the user through the experience) and lead to less reliable measurements [15]. Thus, if we can keep the user in VR during the subjective measurement period of the study, we could reduce the negative effects of leaving the virtual environment. Here is the point that concept of in-VR questionnaires can be defined to provide the possibility of doing questionnaires inside a VR application without leaving the immersive experience to reduce these errors.
In recent years, attention to in-VR questionnaires has increased, and several researchers have used them in their studies, e.g., [15,35,37,38,39,40,41]. [35] reports that there is no significant difference between the results of the in-VR and out-VR questionnaire in the case of presence. A similar result is also reported by [15], which confirms the previous results. Although [35] showed that embedding the questionnaire in the VR can improve the consistency of the results compared to out-VR, [15] found no significant difference. This inconsistency in the results leaves the discussion of the influence of the in-VR questionnaire design on the reported results open for further investigation.
In addition, there is no clear guide for implementing in-VR questionnaires considering UI and interaction systems. Although 3D UI design has great potential to use in VR applications [42], most implemented ones replicate the UI of traditional (non-immersive) desktop experiences. On the other hand, the consistency of the interaction method with the type of content in VR can lead to a better user experience [43]. It may also reduce the time needed to interact with the UI, which can be important for long questionnaires in VR. Alexandrovsky et al. [15] reviewed 123 articles from 2016 to 2019 that addressed the use of questionnaires in VR studies and, in particular, in-VR questionnaires. They identified 15 articles that used in-VR questionnaires. These embedded questionnaires used different presentation methods such as heads-up display, world-based UI, and body-based UI. Their results show diversity in implementing in-VR questionnaires and a lack of comprehensive study.
In this paper, we aim to develop a taxonomy for the design of in-VR questionnaires. In recent years, there are articles that address the review or implementation of in-VR questionnaires, such as the study by Alexandrovsky et al. [15]. While these studies use literature review and/or implementation of in-VR questionnaires, they do not introduce a clear term for these UI elements. The lack of a general term may lead to less use of In-VR questionnaires. In this study, we propose a general term for the user interface of these questionnaires after reviewing the collected literature. Moreover, the study by Alexandrovsky et al. [15] only considered the traditional user interface for In-VR questionnaires (the common user interface adopted from desktop applications). However, in this review, we expand the UI design options by considering intradiegetic and 3D UI elements, e.g. [21,44]. Accordingly, we identified two dimensions for this taxonomy: UI and interaction type. We reviewed the screened articles to find a relevant application of these two dimensions to use embedded questionnaires in VR. This taxonomy can help the HCI community to choose an appropriate UI and interaction when designing an in-VR questionnaire based on their application.

3. Methodology

We conduct a literature review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [45] for the paper screening procedure. We extracted the literature for this review using Scopus (one of the largest abstract and citation databases of peer-reviewed literature that can search in various libraries such as ACM, Springer, and ScienceDirect) and IEEEXplore (as a common database for computer science and technology-related articles). Our search terms in both databases are: (“in-VR” OR “in virtual reality”) AND (“Questionnaire” OR “Survey”). The first author of this paper collected the literature from the defined databases. We consider a data range for this literature gathering from 01.01.2014 to 31.12.2021 that leads to 1109 records (976 from Scopus and 133 from IEEEXplore). We can summarize our exclusion criteria as follows:(1) The papers that were published before 2014, (2) did not have an appropriate format (included only an abstract or preview), (3) duplicates, (4)papers that were written in languages other than English, (5) papers that were described experiences without HMDs, (6) papers to which we did not have access We included articles that had “in VR questionnaire” or “inVRQ” (common terms for embedded questionnaires in VR) in their title or keywords, then searched for them in the abstract, and finally searched for related content to in-VR questionnaire in the full text. We performed the screening process manually and did not use any automation tool. We performed an informal quality assessment of the contribution content focused on the application of the in-VR questionnaire and the implementation methods. As reviewing this topic is rather new, for this topic, we utilized inductive thematic analysis to explore new insights and perspectives on its taxonomy and application. Accordingly, from 734 articles published between 2014 and 2021 (after the release of consumer-based VR device to the time of data collection), we selected 25 articles for further discussion in this paper that considered either the in-VR questionnaire as an assessment tool or as a topic for their study. Among these articles, nine reported on user studies using in-VR questionnaires, one developed only a toolkit without a user study, and two were review papers. In Table 1, we summarize the reviewed articles in the literature considering their topic, the number of participants, and the implemented questionnaires.

4. Results

Our review indicates, most articles considered mouse pointer interaction as one of the typical interaction types in VR. They used a traditional 2D user interface (similar to the user interface in out-VR) for in-VR questionnaires. Most articles used standard or custom questionnaires with multiple-choice responses, such as a 5-point Likert scale. We have organized the literature we gathered into three main themes: questionnaires, design, and toolkit. Articles that focused on questionnaires mainly discussed the use of in-VR questionnaires and compared them to out-VR questionnaires. Papers that focused on design discussed various considerations for designing in-VR questionnaires. Finally, some papers provided or recommended a toolkit for implementing in-VR questionnaires.
Questionnaires: One of the main characteristics of VR is a high level of immersion that leads to a sense of presence. According to Slater, [62] immersion can be defined as the objective perception of a virtual experience that is influenced by device technology. Higher sensory displays and tracking technology that lead to a more consistent experience with real sensory perception can lead to higher immersion. On the other hand, presence is the subjective feeling of "being" in a virtual environment. Given the same level of immersion, we can expect different degrees of presence depending on the user’s response. Therefore, measuring the sense of presence is crucial for VR research and development. Completing the questionnaire outside the virtual environment may lead to disruption of the feeling of presence because of time consumption and disorientation, it is essential to compare the effect of questionnaires in VR and outside VR on user experience and outcomes.
Schwind et al. [35] examined the effects of embedding questionnaires in a virtual environment on participant response. A total of 36 participants took part in the study by playing a first-person shooter game in VR. They found that when questionnaires are used in the virtual world, the variance of the measurement can be kept constant, but this is not the case when questionnaires are used outside the virtual world. This difference could be due to the fact that the probability of a “Break In Presence” (BIP) is lower when using in-VR questionnaires. BIP can be defined as a moment when the sense of presence is disturbed and the user becomes aware of the real environment [63]. A sudden change of context or taking off the headset to answer the questionnaire can lead to BIP, which can bias the assessment results [9]. The effects of BIPs on performance and presence may vary depending on the complexity and workload of the VR tasks, as well as the type of VR environment (e.g., learning, productivity, therapy). Embedding questionnaires directly into the VR environment can contribute to the consistency of self-reports on presence, but the extent and influence of BIPs on the VR experience need to be further investigated. A study by Tamaki and Nakajima [58] shows that some items of the IPQ (e.g., “I am in the VR space” and “I am involved in the virtual environment”) decrease when users experience the transition between VR and out-VR questionnaires. Putze et al. [9] discussed the effects of in-VR questionnaires on BIPs compared to out-VR questionnaires. They recorded biosignals from 50 participants in VR games with two environmental designs, one realistic and one simplified. The results of this study show that both in-VR and out-VR questionnaires can cause BIPs. However, the in-VR questionnaires produce less intense BIPs and may result in more reliable responses.
Although questionnaires are well-known tools for assessing presence in virtual environments, a recent study by Graf and Schwind [50] shows inconsistencies between measurement and main results. This finding is in agreement with Alexandrovsky et al. [15] findings that presence questionnaires are not sufficient to measure BIPs. They recommend measuring presence in the virtual environment, either behaviorally or physiologically. The study by Graf and Schwind [50] also shows that some results are influenced by the environment in which the questionnaires are placed and the type of interaction with them. Accordingly, they recommend a revision of the presence assessment tool in VR and show the importance of studying the influence of the design of the questionnaire environment on users’ responses. However, this result contradicts the results of Regal et al. [18] study on the influence of using a special environment for questionnaires or using questionnaires as a scene object. Regal et al. [18] report that they could not find any influence of the questionnaire environment on the result of the presence questionnaire.
Design: Using questionnaires in VR can be a research topic in itself focused on the design and development of these virtual elements. Here, the lack of clear guidelines and standards for 3D UIs leads to many different UIs for questionnaires in the literature. Alexandrovsky et al. [15] conducted user studies to evaluate different design choices for in-VR questionnaires. They began the study by surveying VR experts about their attitudes toward VR questionnaires. The general response was positive, but there was no standard method for presenting questionnaires in the virtual environment.
Moreover, in contrast to the positive tendency of the experts to embed questionnaires in VR, there were also some arguments against it. These arguments are mainly related to the difficulties in designing/implementing questionnaires in VR and overloading the participants during the survey, which may lead to the bias of the results. The authors then discussed a conducted user study in which they designed two positions (world-anchored and body-anchored) for questionnaire placement and implemented two interaction types (pointer and trackpad). Their results show that placing the questionnaire in a fixed position in the virtual environment with pointer interaction works better than the other options. This result is consistent with the results of Safikhani et al. [21], where the authors compared two alternatives for designing questionnaires in VR with those outside of VR. They considered two world-anchored objects for displaying questionnaires in the virtual environment and called them 2D and 3D layouts. The first alternative, the 2D layout, is a monitor on the wall that displays the question text and allows the user to select the answer with a pointer. In the second case, the 3D layout, they designed a virtual device with multiple monitors and handles to display the text and answer choices. Users can select, accept, and change the answer using the handles via VR grip interaction. Their results suggest that most participants prefer the in-VR design of questionnaires. However, task load is higher compared to out-VR, which is consistent with [15] findings.
Even when questionnaires are embedded in the virtual environment, the consistency of the design with the environment can affect the user experience. Wagener et al. [44] investigated the impact of the extent of integration and consistency of the questionnaire with the virtual environment on the user experience and the duration of the entire study. They considered four design alternatives for the questionnaire UI and categorized them as extradiegetic and intradiegetic. Their results suggest that an intradiegetic design approach can lead to higher presence, perceived user experience, and expert acceptance. However, the time required to complete the questionnaires is higher. A similar finding is also reported by Safikhani et al. [21] that questionnaires with 3D layouts are more time-consuming.
Toolkit: Considering the additional effort required to implement questionnaires in the virtual environment, researchers have developed toolkits to facilitate this process. Feick et al. [49] has developed a customizable toolkit to quickly implement a basic user interface, similar to the traditional desktop user interface, in VR. Researchers can use either a set of standard predefined questionnaires, such as NASA TLX and SUS, or custom questionnaires with JSON files. Safikhani et al. [21] accompanied their research with an immersive questionnaire toolkit. With their toolkit, researchers can implement standard or custom questionnaires, either based on XML files or by importing from the Limesurvey website (limesurvey.org). With this toolkit, evaluation results can later be exported in XML file format or uploaded to the Limesurvey website for post-processing. We found also articles using in-VR questionnaires as an evaluation tool for their studies. For example, Wijnen et al. [60] simulate a virtual museum tour where participants encounter interacting robots. They used a custom in-VR questionnaire to assess the feeling of interacting with robots in VR and the perceived scene of presence. In this study, the in-VR questionnaire was presented as a 2D UI object fixed at a specific location in the scene. Participants can interact with the questionnaire through pointer interaction. The same type of interaction is used by Krekhov et al. [52], but a full-screen 2D UI was used to present NASA TLX and a custom questionnaire.

4.1. Classification

Based on the reviewed literature, we can categorize the UI for implemented questionnaires in VR into two general categories: 2D UI and 3D UI (see Figure 1). This distinction is related to the diegetic design of UI (i.e., 2D-UI can be counted as an interface in non-diegetic space and 3D-UI can be defined as a diegetic UI because it appears in the game’s context). According to these two types, developers can implement the in-VR questionnaire with different positioning approaches: full-screen, world-anchored, and body-anchored.
The full-screen approach is similar to the traditional desktop experience (i.e., it pushes the game aside and forces the user to see and interact with the questionnaire). This presentation can reduce the user’s sense of presence by disconnecting them from the virtual context. The world-anchored representation can be defined as a UI element in the virtual environment that is tied to a part of the scene. It can be fully fixed (the user must align to read the content) or rotate around an edge of the UI to align with the user’s view; an example of this positioning approach can be found in a study by Rzayev et al. [64,65]. Body anchor positioning connects the questionnaire UI to a part of the virtual character (mainly the hand). The advantages of this approach are the availability of the questionnaire in all environments and the possibility to change the position of the UI on the go.
Alexandrovsky et al. [15] used this approach to find suitable design options for in-VR questionnaires. There is also a side approach that is mainly a combination of world anchor and body anchor to take advantage of both cases. In this case, the users can trigger the user interface of the questionnaire anywhere in the virtual environment according to their preferences, but the questionnaire functions as an object anchored in the world. An example of this hybrid positioning option is described by Safikhani et al. [66] for an in-VR inventory and menu system.
We can summarize the benefits of each UI representation as the following:
2D-UI | Fullscreen→Benefits: visibility of the text, focus on the questionnaire, familiar interface (similar to the traditional desktop UI), Drawbacks: may reduce the sense of presence, detach the user from virtual experience.
2D-UI | Body-anchored→Benefits: freedom in positioning and spawning, familiar interface (similar to the tradition desktop UI), Drawbacks: more physically demanding, instability of hand during interaction, distinguishable from the rest of the scene (i.e. inconsistent with other virtual elements in the experience)
2D-UI | World-anchored→Benefits: easy to interact, easy to read, familiar interface (similar to the tradition desktop UI), Drawbacks: limited to the specific area of the scene, hard to read from distance, distinguishable from the rest of the scene)
3D-UI | Body-anchored→Benefits: freedom in positioning and spawning, consistency with other virtual elements in the environment, Drawbacks: more physically demanding, instability of hand during interaction)
3D-UI | World-anchored→Benefits: easy to interact, easy to read, consistency with other virtual elements in the environment, Drawbacks: limited to the specific area of the scene, hard to read from distance, more physical demanding than 2D-UI due to the interaction type)
The type of interaction in VR is an inseparable part of the virtual experience and is related to the type of interface. Over the years, several studies have been conducted to develop and improve interaction systems in VR. Interaction type can affect the entire user experience and change the way the user perceives this immersive environment.
Hepperle et al. [43] investigated the impact of interaction type on user preferences and performance. Their quantitative user study considered the following three main types of user input: 2D, 3D, and speech. In their study, the 2D interface is represented by displaying 2D icons to select colors or move objects, the 3D interface is a direct interaction with virtual objects (inserting a paintbrush into a paint bucket or moving an object by grabbing it), and the speech interface can perform prior actions via voice commands. The results show that the type of interaction can affect the user experience. A 3D UI and corresponding interaction are more appropriate when perceived presence is a critical issue. A 2D interface can be chosen when accurate or fast interaction is required. Voice input is easy to learn and suitable for long text input.
Considering the study of Hepperle et al. [43], the content of the VR application should be considered when deciding on UI and interactions. Accordingly, the application for in-VR questionnaires should be based on the content of the questionnaire. For example, if presence perception is a major part of the questionnaire, 3D UI and interaction can be a solution. In Figure 2, we categorize the possible interaction types for in-VR questionnaires. The classification is based on the input device and the resulting interaction. Some of these interaction types are not used in the studies discussed in this paper and can be a topic for future research and development. Furthermore, different input devices can be used for an identical resulting interaction (e.g., controller input and hand tracking can be used for pointer interaction), but with different usability and corresponding user experience. We can summarize the benefits and drawbacks of these types of interactions as the following:
Trackpad→Benefits: accurate and easy to use, similar to traditional input system using gamepad or keyboard, Drawbacks: may lead to distraction from virtual environment, difficult to manipulate when there are so many items to choose from.
Pointer→Benefits: easy to implement and find template for implementation, similar to traditional input system using mouse, Drawbacks: instability of the hands during interaction, physically demanding for long questionnaire.
VR touch→Benefits: may lead to a higher perceived sense of presence, easy to interact, Drawbacks: limited interaction range (user’s virtual hand should touch the interaction area).
VR grip→Benefits: may lead to a higher perceived sense of presence, consistent with the rest of virtual scene, Drawbacks: limited interaction range (user should grab the interactive object), can be slower than others and be physically demanding.

5. Discussion

RQ1: As shown in the Results section, there is no explicit naming convention for In-VR questionnaires. The lack of an exact keyword or abbreviation makes it difficult to research and reference this assessment tool. Accordingly, researchers cannot easily gather information about recent developments in this field. Therefore, we recommend using the term “inVRQ” as an abbreviation and keyword for in-VR questionnaires in future studies.
RQ2: In most studies, a pointer was used to interact with the questionnaire UI. On the other hand, based on several literature reports (e.g., Safikhani et al. [21] and Alexandrovsky et al. [15]), it seems that in-VR questionnaires may have higher task requirements. Here, the question may be: Is there a relationship between interaction type and task demand? Accordingly, investigating the influence of interaction type on task demands could be an important future research topic. Although the studies mainly used traditional 2D user interfaces, using a more consistent design approach for in-VR questionnaires may lead to a higher sense of presence, as discussed by Wagener et al. [44]. Considering an appropriate user interface for questionnaire presentation, as well as improving interactions in VR, may lead to increased use of in-VR questionnaires. The majority of the presented questionnaires were limited to multiple-choice questions. Although these types of questionnaires are typical in studies, the idea of implementing free-form questionnaires remains important. Incorporating a free-form questionnaire can be helpful, especially after the experience when users are required to provide their personal feedback. Discouraging the implementation of free-form questionnaires may be related to the difficulty of typing in VR. Here, it will be of interest to find a suitable solution for another type of user input.
RQ3: The ability to conduct VR studies remotely may facilitate the study procedure. In addition, the increasing number of consumer HMDs will likely make remote evaluation more feasible. In this case, using the in-VR questionnaire as part of a distance learning package may be a solution to facilitate VR studies and increase the number of participants. Based on our findings, we can recommend the following points for future studies: (1) Investigating the influence of interaction type on task load during the filling of in-VR questionnaires (2) Define and implement a standard design for in-VR questionnaires, considering the consistency with the virtual environment (3) Study on appropriate user input for different kinds of questions, such as free-form ones. (4) Design and development of an all-in-one toolkit for importing, interacting, and exporting in-VR questionnaires with the possibility of running remote studies
As an overview, in-VRQ questionnaires can improve the overall user experience and consistency of results by reducing BIPs and eliminating the interruptions caused by switching between the real and virtual worlds. They can help in conducting the study remotely. However, the development of an in-VR questionnaire is less relevant due to the implementation challenges and difficulties in finding relevant literature. We believe that this categorized structure for interaction types and UI design options could help developers during implementation. On the other hand, our recommendation of a general term, inVRQ, could help future studies refer to these questionnaires and lead to clearer results in future literature collections.

6. Conclusion

In VR research, measuring parameters related to the VR experience, such as presence, can be affected by an interruption in the experience that may lead to inaccuracy in the results. According to the literature, using questionnaires embedded in VR can lead to less break in presence. In this study, we conducted a literature review on the use of In-VR questionnaires. We reviewed a total of 25 articles from 2014-2021 in which an In-VR questionnaire was used either as an assessment tool or as a topic for study.
This study can be extended by collecting literature from more databases. We also did not perform a quality assessment for the content of the reviews because we were only interested in the application of the in-VR questionnaires. The use of quality assessment tools, such as JBI, may help future research to ensure the quality of reported studies. The results of studies on the application of in-VR questionnaires in comparison to the out-VR type indicate a positive response of participants towards in-VR ones. In this case, the consistency of in-VR questionnaire design with the virtual environment from both interaction and UI may lead to a higher perceived sense of presence. However, comparing the task load in in-VR and out-VR questionnaires shows higher values in the case of in-VR ones. Accordingly, studies on the influencing factors on increasing in-VR experience task load can be investigated to improve the usability and acceptability of this type.
The difficulties in the design and implementation of in-VR questionnaires can be a barrier to an increase in their application. Considering this point, the development of comprehensive toolkits that makes the implementation of it faster and simpler can lead to more acceptance among researchers. In addition, according to the increased number of consumer-based HMDs in recent years, developing a toolkit with the possibility of remote evolution can help studies to have more participants. These kinds of toolkits can also be helpful in a situation where accessing users is not possible or limited, such as the current COVID-19 pandemic. In this case, an all-in-one VR questionnaire package may be used more in research communities. This taxonomy of in-VR questionnaires presented in this paper may provide a useful overview of common ways to present and interact with questionnaires in VR.

References

  1. Nascimento, A.M.; Queiroz, A.C.M.; Vismari, L.F.; Bailenson, J.N.; Cugnasca, P.S.; Junior, J.B.C.; de Almeida, J.R. The role of virtual reality in autonomous vehicles’ safety. 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 2019, pp. 50–507.
  2. Goedicke, D.; Li, J.; Evers, V.; Ju, W. VR-OOM: Virtual reality on-road driving simulation. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–11.
  3. Safikhani, S.; Keller, S.; Schweiger, G.; Pirker, J. Immersive virtual reality for extending the potential of building information modeling in architecture, engineering, and construction sector: systematic review. International Journal of Digital Earth 2022, 15, 503–526. [Google Scholar] [CrossRef]
  4. Safikhani, S.; Pirker, J.; Wriessnegger, S.C. Virtual Reality Applications for the Treatment of Anxiety and Mental Disorders. 2021 7th International Conference of the Immersive Learning Research Network (iLRN). IEEE, 2021, pp. 1–8.
  5. Pottle, J. Virtual reality and the transformation of medical education. Future healthcare journal 2019, 6, 181. [Google Scholar] [CrossRef] [PubMed]
  6. Sidani, A.; Dinis, F.M.; Sanhudo, L.; Duarte, J.; Baptista, J.S.; Martins, J.P.; Soeiro, A. Recent tools and techniques of BIM-based virtual reality: A systematic review. Archives of Computational Methods in Engineering 2021, 28, 449–462. [Google Scholar] [CrossRef]
  7. di Lanzo, J.A.; Valentine, A.; Sohel, F.; Yapp, A.Y.; Muparadzi, K.C.; Abdelmalek, M. A review of the uses of virtual reality in engineering education. Computer Applications in Engineering Education 2020, 28, 748–763. [Google Scholar] [CrossRef]
  8. Barnard, D. History of VR - timeline of events and Tech Development, 2022.
  9. Putze, S.; Alexandrovsky, D.; Putze, F.; Höffner, S.; Smeddinck, J.D.; Malaka, R. Breaking the experience: Effects of questionnaires in vr user studies. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–15.
  10. Das, S.; Maiti, J.; Krishna, O. Assessing mental workload in virtual reality based EOT crane operations: A multi-measure approach. International Journal of Industrial Ergonomics 2020, 80, 103017. [Google Scholar] [CrossRef]
  11. Hertweck, S.; Weber, D.; Alwanni, H.; Unruh, F.; Fischbach, M.; Latoschik, M.E.; Ball, T. Brain Activity in Virtual Reality: Assessing Signal Quality of High-Resolution EEG While Using Head-Mounted Displays. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 970–971.
  12. Abeele, V.V.; Spiel, K.; Nacke, L.; Johnson, D.; Gerling, K. Development and validation of the player experience inventory: A scale to measure player experiences at the level of functional and psychosocial consequences. International Journal of Human Computer Studies 2020, 135. [Google Scholar] [CrossRef]
  13. Schubert, T.; Friedmann, F.; Regenbrecht, H. The experience of presence: Factor analytic insights. Presence: Teleoperators & Virtual Environments 2001, 10, 266–281. [Google Scholar]
  14. Ryan, R.M.; Rigby, C.S.; Przybylski, A. The motivational pull of video games: A self-determination theory approach. Motivation and emotion 2006, 30, 344–360. [Google Scholar] [CrossRef]
  15. Alexandrovsky, D.; Putze, S.; Bonfert, M.; Höffner, S.; Michelmann, P.; Wenig, D.; Malaka, R.; Smeddinck, J.D. Examining design choices of questionnaires in vr user studies. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–21.
  16. Brühlmann, F.; Mekler, E.D. Surveys in Games User Research. In Games User Research; Oxford University Press: Oxford, 2018. [Google Scholar] [CrossRef]
  17. Harris, D.J.; Buckingham, G.; Wilson, M.R.; Vine, S.J. Virtually the same? How impaired sensory information in virtual reality may disrupt vision for action. Experimental brain research 2019, 237, 2761–2766. [Google Scholar] [CrossRef]
  18. Regal, G.; Voigt-Antons, J.N.; Schmidt, S.; Schrammel, J.; Kojić, T.; Tscheligi, M.; Möller, S. Questionnaires embedded in virtual environments: reliability and positioning of rating scales in virtual environments. Quality and User Experience 2019, 4, 1–13. [Google Scholar] [CrossRef]
  19. Steinmaurer, A.; Sackl, M.; Gütl, C. Engagement in In-Game Questionnaires-Perspectives from Users and Experts 2021. pp. 1–7.
  20. Frommel, J.; Rogers, K.; Brich, J.; Besserer, D.; Bradatsch, L.; Ortinau, I.; Schabenberger, R.; Riemer, V.; Schrader, C.; Weber, M. Integrated questionnaires: maintaining presence in game environments for self-reported data acquisition 2015. pp. 359–368.
  21. Safikhani, S.; Holly, M.; Kainz, A.; Pirker, J. The Influence of in-VR Questionnaire Design on the User Experience. Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, 2021, pp. 1–8.
  22. Young, L. International journal of advanced smart convergence 2021, 10, 72–80. 10.
  23. Hertel, J.; Karaosmanoglu, S.; Schmidt, S.; Bräker, J.; Semmann, M.; Steinicke, F. A Taxonomy of Interaction Techniques for Immersive Augmented Reality based on an Iterative Literature Review. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2021, pp. 431–440.
  24. Kim, Y.M.; Rhiu, I.; Yun, M.H. A systematic review of a virtual reality system from the perspective of user experience. International Journal of Human–Computer Interaction 2020, 36, 893–910. [Google Scholar] [CrossRef]
  25. Motejlek, J.; Alpay, E. Taxonomy of virtual and augmented reality applications in education. IEEE transactions on learning technologies 2021, 14, 415–429. [Google Scholar] [CrossRef]
  26. Eeles, P. Capturing architectural requirements. IBM Rational developer works 2005. [Google Scholar]
  27. Bhowmick, S.; Sorathia, K.; Chandra Kalita, P. Understanding Gesture Performance for Object Selection in VR: Classification and Taxonomy of Gestures in HCI. In India HCI 2021; 2021; pp. 90–93.
  28. Ruscella, J.; Obeid, M.F. A Taxonomy for Immersive Experience Design. 2021 7th International Conference of the Immersive Learning Research Network (iLRN). IEEE, 2021, pp. 1–5.
  29. Lazar, J.; Feng, J.H.; Hochheiser, H. Research methods in human-computer interaction; Morgan Kaufmann, 2017.
  30. Wobbrock, J.O.; Kientz, J.A. Research contributions in human-computer interaction. interactions 2016, 23, 38–44. [Google Scholar] [CrossRef]
  31. Shneiderman, B. Claiming success, charting the future: micro-HCI and macro-HCI. interactions 2011, 18, 10–11. [Google Scholar] [CrossRef]
  32. Field, A.; Hole, G. How to design and report experiments; Sage, 2002.
  33. Rosenthal, R.; Rosnow, R.L. Essentials of behavioral research: Methods and data analysis; 2008.
  34. Choi, B.C.; Pak, A.W. Peer reviewed: a catalog of biases in questionnaires. Preventing chronic disease 2005, 2. [Google Scholar]
  35. Schwind, V.; Knierim, P.; Haas, N.; Henze, N. Using presence questionnaires in virtual reality. Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–12.
  36. Knibbe, J.; Schjerlund, J.; Petraeus, M.; Hornbæk, K. The dream is collapsing: the experience of exiting VR. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–13.
  37. Kang, H.; Lee, G.; Kwon, S.; Kwon, O.; Kim, S.; Han, J. Flotation Simulation in a Cable-driven Virtual Environment–A Study with Parasailing. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–11.
  38. Cao, Z.; Jerald, J.; Kopper, R. Visually-induced motion sickness reduction via static and dynamic rest frames. 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, 2018, pp. 105–112.
  39. Schwind, V.; Mayer, S.; Comeau-Vermeersch, A.; Schweigert, R.; Henze, N. Up to the finger tip: The effect of avatars on mid-air pointing accuracy in virtual reality. Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, 2018, pp. 477–488.
  40. Fernandes, A.S.; Feiner, S.K. Combating VR sickness through subtle dynamic field-of-view modification. 2016 IEEE symposium on 3D user interfaces (3DUI). IEEE, 2016, pp. 201–210.
  41. Wienrich, C.; Schindler, K.; Döllinqer, N.; Kock, S.; Traupe, O. Social presence and cooperation in large-scale multi-user virtual reality-the relevance of social interdependence for location-based environments. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2018, pp. 207–214.
  42. LaViola Jr, J.J.; Kruijff, E.; McMahan, R.P.; Bowman, D.; Poupyrev, I.P. 3D user interfaces: theory and practice; Addison-Wesley Professional, 2017.
  43. Hepperle, D.; Weiß, Y.; Siess, A.; Wölfel, M. 2D, 3D or speech? A case study on which user interface is preferable for what kind of object interaction in immersive virtual reality. Computers & Graphics 2019, 82, 321–331. [Google Scholar] [CrossRef]
  44. Wagener, N.; Stamer, M.; Schöning, J.; Tümler, J. Investigating Effects and User Preferences of Extra-and Intradiegetic Virtual Reality Questionnaires. 26th ACM Symposium on Virtual Reality Software and Technology, 2020, pp. 1–11.
  45. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; others. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Bmj 2021, 372. [Google Scholar]
  46. Bushra, N.; Carruth, D.; Deb, S. A comparative study of virtual UI for risk assessment and evaluation. International Symposium on Visual Computing. Springer, 2018, pp. 226–236.
  47. Cha, S.H.; Koo, C.; Kim, T.W.; Hong, T. Spatial perception of ceiling height and type variation in immersive virtual environments. Building and Environment 2019, 163, 106285. [Google Scholar] [CrossRef]
  48. Degraen, D.; Zenner, A.; Krüger, A. Enhancing texture perception in virtual reality using 3D-printed hair structures. Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–12.
  49. Feick, M.; Kleer, N.; Tang, A.; Krüger, A. The Virtual Reality Questionnaire Toolkit. Adjunct Publication of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020, pp. 68–69.
  50. Graf, S.; Schwind, V. Inconsistencies of Presence Questionnaires in Virtual Reality. 26th ACM Symposium on Virtual Reality Software and Technology, 2020, pp. 1–3.
  51. Kopel, M.; Rutkowska, M. Which Gameplay Aspects Impact the Immersion in Virtual Reality Games? International Conference on Computational Collective Intelligence. Springer, 2021, pp. 159–171.
  52. Krekhov, A.; Cmentowski, S.; Waschk, A.; Krüger, J. Deadeye visualization revisited: Investigation of preattentiveness and applicability in virtual environments. IEEE transactions on visualization and computer graphics 2019, 26, 547–557. [Google Scholar] [CrossRef]
  53. Luong, T.; Lecuyer, A.; Martin, N.; Argelaguet, F. A Survey on Affective and Cognitive VR. IEEE Transactions on Visualization and Computer Graphics 2021. [Google Scholar] [CrossRef] [PubMed]
  54. Marín-Morales, J.; Higuera-Trujillo, J.L.; Greco, A.; Guixeres, J.; Llinares, C.; Scilingo, E.P.; Alcañiz, M.; Valenza, G. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Scientific reports 2018, 8, 1–15. [Google Scholar] [CrossRef] [PubMed]
  55. Oberdörfer, S.; Heidrich, D.; Latoschik, M.E. Usability of gamified knowledge learning in VR and desktop-3D. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–13.
  56. Regal, G.; Schatz, R.; Schrammel, J.; Suette, S. VRate: A Unity3D asset for integrating subjective assessment questionnaires in virtual environments. 2018 Tenth international conference on quality of multimedia experience (QoMEX). IEEE, 2018, pp. 1–3.
  57. Schwind, V.; Knierim, P.; Tasci, C.; Franczak, P.; Haas, N.; Henze, N. " These are not my hands!" Effect of Gender on the Perception of Avatar Hands in Virtual Reality. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017, pp. 1577–1582.
  58. Tamaki, R.; Nakajima, T. Shoot Down Drones with Your Answer, an Integration of a Questionnaire into a VR Experience. Symposium on Spatial User Interaction, 2021, pp. 1–2.
  59. Vieira, J.; Nóbrega, R.; Pereira, V.; Coelho, A.; Jacinto, A.; Morais, C. Knowledge analysis automatic evaluation in virtual reality immersive experiences. 2019 International Conference on Graphics and Interaction (ICGI). IEEE, 2019, pp. 122–129.
  60. Wijnen, L.; Bremner, P.; Lemaignan, S.; Giuliani, M. Performing Human-Robot Interaction User Studies in Virtual Reality. 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 2020, pp. 794–794.
  61. Wölfel, M.; Hepperle, D.; Purps, C.F.; Deuchler, J.; Hettmann, W. Entering a new Dimension in Virtual Reality Research: An Overview of Existing Toolkits, their Features and Challenges. 2021 International Conference on Cyberworlds (CW), 2021, pp. 180–187. [CrossRef]
  62. Slater, M. A note on presence terminology. Presence connect 2003, 3, 1–5. [Google Scholar]
  63. Chung, J.; Yoon, H.J.; Gardner, H.J. Analysis of break in presence during game play using a linear mixed model. ETRI journal 2010, 32, 687–694. [Google Scholar] [CrossRef]
  64. Rzayev, R.; Ugnivenko, P.; Graf, S.; Schwind, V.; Henze, N. Reading in VR: The Effect of Text Presentation Type and Location. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–10.
  65. Rzayev, R.; Woźniak, P.W.; Dingler, T.; Henze, N. Reading on smart glasses: The effect of text position, presentation type and walking. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–9.
  66. Safikhani, S.; Holly, M.; Pirker, J. Work-in-Progress—Conceptual Framework for User Interface in Virtual Reality. 2020 6th International Conference of the Immersive Learning Research Network (iLRN), 2020, pp. 332–335. [CrossRef]
Preprints 108220 g001
Figure 1. The taxonomy of in-VR questionnaire UI
Figure 1. The taxonomy of in-VR questionnaire UI
Preprints 108220 g001
Preprints 108220 g002
Figure 2. The interaction types with in-VR questionnaires
Figure 2. The interaction types with in-VR questionnaires
Preprints 108220 g002
Table 1. An overview of the collected literature, their questionnaires and participants
Table 1. An overview of the collected literature, their questionnaires and participants
Paper Type of study Participants Questionnaires
[15] Literature review, study on UI and interaction 10 IPQ, NASA TLX, UMUX, SUS
[46] Study with application of inVRQ 13 SSQ and SUS
[38] Study with application of inVRQ 22 SSQ
[47] Study with application of inVRQ 43 Custom
[48] Study with application of inVRQ 10 Custom
[49] Toolkit Development - NASA TLX, SUS, SSQ
[40] Study with application of inVRQ 32 Discomfort score
[50] Study on presence in VR 36 IPQ, SUS
[37] Study with application of inVRQ 13 SSQ and SUS
[51] Study on immersion in VR 26 Custom
[52] User study with application of inVRQ 24 NASA TLX, custom
[53] Study with application of inVRQ - -
[54] Study with application of inVRQ 60 SAM
[55] Study with application of inVRQ 13 NASA TLX
[9] Study on the influence of inVRQ 36 IPQ, SUS, PQ
[56] Toolkit Development - -
[21] Study on inVRQ UI design 16 NASA TLX, IPQ, SUS
[35] Study on the influence of inVRQ 36 SUS, WS, IPQ
[57] Study with application of inVRQ 28 WS and custom
[58] Study on the influence of inVRQ 12 IPQ
[59] Study with application of inVRQ 35 Custom
[44] Study on inVRQ UI design 35 Presence and custom
[41] Study with application of inVRQ 12 PANAS and presence
[60] User study with application of inVRQ 38 Custom
[61] Literature review - -
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated