Now that the minimal viable product for the story has been completed, we can then conduct a usability and playability test to see if there are any issues both with the design and program. This usability test will see if there any issues interacting with the interface, to see if there any level design issues as well as how well the Leap Motion and Oculus Rift DK2 enhance the story.
Before we can conduct the test, we need to plan out what we wish to test and how to then run it to reduce any possible bias in the results. We will also need to plan out how many people we should aim to test it on and how the tasks will be given to the user.
To start, we decided we should test the UI as it involves using the Leap Motion which is a very different way of interacting compared to the standard input for computers. It will allow us to see if the user can quickly pick up the idea of using their hand to interact with the world as well as then using their hand to control the UI.
Another aspect we decided to test was how viable the DK2 is and how it enhances or hinders the users experience. For this, we asked if the text on the UI was visible then if they believed it enhanced their experience or if they experienced any motion sickness.
We also will ask the user how they find the story as well as navigating the world. It can identify weak areas of the level design as a poor level design could mean they miss aspects of the story.
Conducting the test and collecting feedback
At the start of the test, we briefed the user about the overall goal of the test as well as what is expected in terms of feedback from them at the end. They was informed that no help would be offered unless asked for and once a task was complete, we would let know and move them onto the next task if required.
For the test, there would be 2 observers who would make notes about what the user is saying as well any difficulties that we may have noticed the user experience. The use of 2 observers is to reduce the load on just one as well as reduce the chance that something is not missed.
To collect data from the users regarding the test, there are methods we decided to use. One is asking them to complete a questionnaire after each task is completed, this is so that it is still fresh in their mind and can answer with more accuracy compared to having them complete it at a later time where they may have forgot. The other way of obtaining feedback is via observation as noted above.
The task list and setup can be found here: UsabilityTestTaskListAndSetup
The test was conducted over a number of days with a total of 7 participants, of which 2 had tested the story before. For the purpose of this analysis, I shall only be focusing on those who have not tried it before.
Of all our participants, 85% were male and 18-25 which is our main target demographic. 71% of testers play games for more than 10 hours a week.
None of our testers were colour blind and all were right handed. This was an important question to ask as our main interaction with the world is via the right hand so users who have a predominantly left hand might struggle.
71% of the testers has used the Leap Motion before and all had used the DK2. This will lead to some bias in our results as they may be more familiar with the general mechanics of VR games and stories.
2 testers managed to reach the cafeteria area of the story where the first branch would take place. Their main reasons for reaching this area is that one was thirsty as the other always goes right when given a choice. All found it easy to navigate.
5 testers reached the server room section of the story where the other branch occurs. The main reasons behind this was that they generally always go left or had reached the other destination and wanted to explore the other direction. The testers who generally went that direction did find it harder to navigate.
There was no real logical choice why they want one direction or another as there was not a major compelling reason as reflected in their response. This is should we could look into addressing in the future by having the user want to know more about certain subjects that can only be found by going a certain direction.
As part of the world interaction section, we tested to see how well the testers interacted with the world and if there was any issues doing so. We also covered how well they could use and distinguish elements on the UI.
57% of the testers could read all the text whilst wearing the DK2 whilst 28% couldn’t read any at all. This is partially a limitation of the DK2 due to the low resolution however it could also be caused by the DK2 not being configured correctly for each individual user.
Out of all the testers, 85% interacted with the world without prompt. This is a very good percentage considering the unusual method of interaction however we did have some users confess that they have heard about how the mechanic works ahead of time, leading to a bias.
A common comment made by many testers was they wanted more interactable objects within the world. We observed that most tried to use the vending machine so it’s something we should look into adding interaction with.
Leap Motion Experience
There was a mixed response across all the testers regarding their experience with the leap motion. There was a lot of issues with hands failing to track correctly or not rendering properly. This was also noticed by our observers as well who made notes on how frustrated the testers got. We also got one feedback where the tester did not like that it was a hybrid control system between Leap Motion and keyboard/mouse. Designing to cater for a Leap only experience, one concern we had with going Leap only is arm fatigue from needing to hold your hands up a lot which is why we discarded it.
Building upon earlier sentiments, a lot of users did not fully utilise the Rift, more so once they realised how they could control their character. Most of the participants did not know how to properly move their character with the majority initially thinking that their character will move forward with the direction their head is pointing.
Testers enjoyed the unique AR style UI as they felt it was very immersive. There was also a few bugs that was reported or noticed by our observers which have now been fixed.
One feedback we got was with regards to the level design in that the hallways were too generic and boring, not offering them a good enough indication of where they were.
Based upon the feedback from the usability test, a number of changes were made to reduce frustration for the user as well as make more logical sense.
One of the major changes was the way the UI worked with the text size being increased. How it also appears and disappears was also changed so it would linger around longer and retrack the hand to compensate for issues with the Leap Motion.
Signs were also added to the corridors to help guide the users to certain destinations and prevent them from getting lost. Tutorial signs were also added to the starting room so that users could see how to interact with the world as well as move about.
There are some more changes we need to investigate such as changing how the player moves around including turning their body. Unfortunately doing such a change will require a lot of research and more usability tests which we did not have time to conduct.
As just mentioned, there are some areas that will take more time to investigate into how to solve such as the interaction. However I do have one idea myself on how to solve this based upon other VR experiences and how they handle it.
Instead of having the user move around with keyboard/mouse or gamepad, they could instead teleport between the different rooms. This could play in with our idea that you are an AI and by you “teleporting”, it’s more travelling along different data lines to different data centres. There you could interact with the objects around and next to you and complete puzzles to unlock more information.
This would solve some of the locomotion issues and allow an almost DK2/Leap Motion only experience.
Overall our usability test gave us a lot of good feedback that allowed us to make numerous changes to the story. The way in which we conducted the test was good as we felt it allowed us to see how users would naturally react however some users did have prior knowledge of how some of the systems worked even though they had never tried it before. A second usability test is currently being conducted to allow us to see if our changes have had any improvements.