What are the best VR user testing practices?

There are a multitude of ways to research and evaluate VR experiences. The first question any researcher or developer needs to ask is what is the information they would like and what are you going to do with it. This will determine what tools or approaches are taken and when to capture the information. If the aim is to publish the finding in peer reviewed journals then selecting validated tools or undertaking formalised tool evaluations will help with this approach. If its more for iterative design processes and user validation purposes perhaps a less structured qualitative interview and observational approaches could be preferable.

Once you have determined what information you want and for what purpose then you can make more informed decisions about what methods and tools should be used. Previously validated tools can bring a level of credibility to findings but also may not be entirely fit for purpose. If designing bespoke tools for data collection ensure outcomes align with the tool you are creating and the method of data capture. Educational design theory can be invaluable in designing assessment methods which have a high level of reliability and validity for the intended purpose.

In addition to careful preparation of users and the testing environment, adherence to Institutional Review Board (IRB) guidelines is crucial for ensuring ethical considerations are met in user testing for VR. IRBs help safeguard participants' rights, privacy, and well-being throughout the testing process. IRB approvals also ensure that proper consent procedures are followed, risks are minimized, and any potential ethical concerns are addressed before conducting the tests. Integrating IRB oversight into the planning and execution of VR user testing promotes #ResponsibleXR research practices and enhances the credibility of study findings.

Preparation for the users is essential to ensure accurate research findings. The level of preparation however can markedly vary depending on the purpose of the research being undertaken. If its about the streamlined and intuitive the user experience is but not the content of the experience then perhaps there would be less focus on the controls of the experience. If the converse was true and it was researching the efficacy of the experience to teach a skill or process in VR then ensuring significant onboarding of the VR experience, controls and flow will be essential to ensure its not limitations in the technology impacting the findings, rather than the efficacy of learning that skill in VR.

When conducting VR tests, choose between targeting newcomers for insights on usability or experienced users for data on engagement and retention. Tailor setup, tutorials, and guidance to fit your target group, ensuring you capture relevant data. Remember, user familiarity with VR varies, so customize the experience to gather meaningful feedback.

During testing, you should aim to capture as much relevant data as possible. Ideally, this should be done in a way that does not break the sensation of presence experienced by the user in VR. One technique is to record the screen during testing (and possibly a 3rd-person perspective of the user in real life). Then use the recorded video for a collaborative analysis, where the user retrospectively reflects on their experience. This method avoids disrupting the user during testing, while allowing time for more in-depth analysis afterwards. Additionally, it provides an opportunity to see discrepancies between what the user thinks they did and what they actually did, which can lead to interesting insights!

Research for the sake of research is a waste of both your users and your own time. It is essential that if you are conducting any form of research or evaluation you are prepared and open to responding to the findings. In some cases perhaps the limitations of the technology chosen or decisions made may mean there is little which can be done to respond to the findings. In this instance perhaps there needs to be a realignment of where the experience exists on the learning pathway or inclusion of other learning materials around the experience. It is very easy as the designer of these experiences to overlook things we have simply gotten used to. Your end user evaluations are your best view into real world usage and must be treated as such.