Assignment 5 Report

I. Introduction / Overview

• Screen capture:

We will release a VR product designed to provide physical and mental assessment for the elderly. Continuing on from previous development, basic tests of visual memory, reaction, and flexibility have been implemented to allow for a more comprehensive assessment of a user’s ability.

Pitch

Are you an elderly person looking to maintain physical and cognitive functions such as hand-eye coordination, memory, flexibility, and motor skills for everyday tasks such as picking up objects? This application tests all these abilities with VR in the comfort of your own home!

II. Technical Development

The Oculus Quest VR headset, touch controllers, and the Unity engine were used to develop the application.  It consists of four tests (kitchen, memory, flexibility, reaction) in two settings (indoor house, outdoor park).

In the kitchen, the user can answer multiple choice questions displayed on a canvas.  A raycast is sent from the right hand, allowing the user to interact with dropdown UI menus by pressing the trigger on the back of the controller.  For each option in the set sequence (banana, orange, then chocolate Santa) that is correctly chosen, text will appear to indicate that the answer is correct.

A main accessibility feature of the outdoor park scene is the implementation of voice commands, which can be used to move the user between different locations.  This movement system uses the Windows Speech keyword listener in combination with Unity’s AI NavMesh, a system that allows objects to automatically navigate their way to specific transform coordinates in a scene.  The speech recogniser listens for the commands listed in the movement menu, and upon recognising a command it will signal the NavMesh agent to make its way to the corresponding destination.  In an attempt to reduce the chance of the listener not registering an input, it also accepts homophones such as “won”, “too” and “for”.  Following usability testing, the voice commands were updated again as detailed in Section V.  

The memory test also uses these voice inputs.  Upon arriving at this test (entering a detection trigger) a random selection of mushroom, bird and rabbit game objects will be made active in the park.  After being visible for six seconds, the objects disappear and a randomly chosen question will be presented, asking how many of a specific object there was. Voice recognition is used to take the user’s answer, and then the program reveals the correct answer and saves the user’s score.

The arms test is designed to help an elderly user test their flexibility. This test consists of 11 cube game objects positioned in front of the user, and the objective is to touch all of the cubes. When the trigger zone on either of the controllers touches a cube, a function runs to make the cube's material green and  increase the score variable of a separate management script. The arms test also works with voice recognition: if the user calls out the word “result” a function is called to activate a canvas that shows the number of cubes touched.

The VR application also includes testing for hand-eye coordination which requires the user to catch a certain amount of falling objects that are spawned at a random position near the user. To make this test more natural and intuitive to elderly people, the Oculus hand tracking feature was implemented into this scene, which allows users to freely use their hands to make grabbing and pinching hand motions to catch the random falling objects without the use of a VR controller. 

III. 3D Content 

Kitchen:

 The kitchen environment, where the user does activities such as grabbing the fruits (general kitchen models: Brick Project Studio 2022) 

Grabbable items: orange (Schindler 2020), banana (danielayala123 2020) and chocolate Santa (3dfood 2020)

Menu found in the kitchen which explains the task instructions

The section of the indoor environment opposite to the kitchen, where the foods are placed. Assets by Brick Project Studio (2022)

Outdoor Park:

The outdoor environment where the memory, flexibility, and reaction tests are done (house: Brick Project Studios 2022) 

The memory test uses three kinds of 3D models: mushrooms (Adventure Forge 2018), birds (dinopunch 2018) and rabbits (Niwashi Games 2019). A random selection of the models is displayed to the user, and they must memorise the number of each object to answer a randomly chosen question.

Decorative items: Realistic tree and park bench model by Pixel Games (2016, 2015). These models are used to add a level of realism to the outdoor scene.

Reaction Test:

In the reaction tests, the user will use hand tracking to catch the cubes objects that randomly fall from above the user by using grabbing and pinching hand motions.

Arm Flexibility: 

In the flexibility test, there are 11 cubes which turn green when the user touches them. All the cubes are positioned so that it shouldn't be too hard for the elderly to perform the test. 

IV. Usability Testing 

Design and Plan

The first step of the usability testing was creating an online survey.  Google Forms was used, as it allows data to be quickly aggregated and displayed as graphs.  The survey contains 19 questions covering each of the four tests implemented in the application. Participants were asked to rate the design of each test on a 1-5 scale to get a snapshot of the general attitude towards each test. Each section then asks specific questions relevant to the technical and design components unique to each test. A final rating scale from 1-10 was included at the end for testers to evaluate the application as a whole. The survey form also contains an instruction script which was read to each tester as they progressed through the application. Please refer to Appendix I (or this link) to access the full survey contents.

Recruitment

Four participants (two students and two relatives) were asked to participate in the usability testing.  This was deemed an appropriate number because the purpose of the testing was not to prove anything with statistical significance, but rather to quickly identify areas of improvement for the application, and by adding more testers we would likely reach a point of diminishing returns for our feedback.  It should be acknowledged that these testers were not in our target age range (65 years+), therefore future testing with the elderly would be necessary to truly evaluate the application’s design. 

Protocol of the testing

Usability testing commenced by reading the initial paragraph of the testing instructions (refer to Appendix I) to give an overview of the application’s purpose. Testers were then given the VR Oculus headset and controllers to test the application. Each tester completed the interactions in the order specified by the testing instructions. Following testing, participants evaluated the application by completing the online survey.

Report and Analysis

 In the figures shown below, a high score indicates satisfaction/agreement, and vice versa for a low score.

Kitchen Test

The questions for the kitchen test were asked to determine whether the different ways of using the controller were intuitive. All participants were satisfied with the menu raycast interaction. They also agreed that the grabbable objects were generally easy to interact with, however their written feedback indicates that specific interactions with the knife are not well implemented, which is what was observed during the testing. One comment also suggests that the Oculus handheld controllers may not be naturally intuitive for the elderly.     

Memory Test

Participants were asked to identify general strengths/weaknesses of the memory test, as well as being asked specifically about voice recognition, as it is most prominently used in this test. The data shows that users were satisfied with the overall design of the test, however the method of giving an answer using voice recognition was not well-received because this feature rarely worked for the testers. Given the low rating, this feedback was addressed as a matter of priority as outlined in section 5.  

Reaction Test

These questions were designed to collect insight on the hand tracking functionality, pacing of the reaction test, and if users found the testing comfortable, to help gauge if the test would be suitable for elderly users. From the responses, most testers were satisfied with the general idea and the implementation of the reaction tests and found that the hand tracking feature was intuitive to use (25% rated 3, 25% rated 4, 50% rated 5).  Half of the testers found that the 3D objects were easy to recognise and 75% of the testers felt that the distance to grab the objects was comfortable. However, many of the testers felt that the rate of the cubes being dropped was too fast, with the cubes falling before the user was ready to catch them, indicating that this would not be suitable for an elderly target audience.  Users also experienced issues where the hand tracking feature did not work for brief moments.

 Flexibility Test

The questions for the flexibility test were aimed at understanding whether the test provided an adequate but not excessive physical challenge. As we did not have elderly participants for this test, users were asked to consider their experience from an older person’s point-of-view. The flexibility test had a positive reception from participants, with unanimous satisfaction towards the simple design. While performing the test, all the users agreed that they had to stretch and make proper body movement in order to complete the test, and most users felt that the positioning of the cubes would be appropriate for the elderly. 

To summarise, the feedback received generally suggests that the tests were well-designed, but the user experience was worsened when technical problems were present in specific interactions.

 V. Addressing the Results of the Usability Testing

Feedback from the kitchen test indicated that the knife was difficult to grab. To address this issue, in future the hand-tracking system could be implemented to make the interaction of grabbing objects smoother. The indoor scene could also benefit from music/sound effects to improve the immersion while grabbing items.

For the memory test, in response to a comment the 3D models have been repositioned to avoid one thing obstructing another.  With the voice recognition, after some experimentation it was discovered that words/phrases with more than one syllable are easier for the program to recognise. Accordingly, when responding to the memory test's question the word “answer” can be said prior to saying the number, e.g. “answer two”. When moving to locations, the commands have been changed to the name of the test, e.g. “memory”, “flexibility”.  This improved the rate of success, however in future the existing voice recognition system will need to be replaced with an Android compatible alternative, as it was discovered that the chosen Windows Speech recogniser is incompatible with Android builds.

The feedback from the flexibility test was positive and users were satisfied with the simple design. In future technical development, hand-tracking could be implemented instead of holding controllers to make it simpler for the elderly to perform this test.

For the reaction test, users said that objects were spawning too quickly, and some thought the objects were hard to see. The changes made to address the issue were to increase the drag of the cubes to reduce the fall speed, and increase the delay between the cubes spawning. These changes will allow the user more time to process the location of the cubes and catch them. The testing also revealed that the hand-tracking can take some time to detect the user’s hands, by which point multiple cubes would have been dropped. To address this issue, the voice command “ready” was added to start the test, allowing users to start the test themselves after the tracking is working.

 VI. References

 Assets 

3dfood 2020, Free 3D santa claus model - TurboSquid 1478073, www.turbosquid.com, viewed 23 October 2022, <https://www.turbosquid.com/3d-models/3d-santa-claus-model-1478073>.

Adventure Forge 2018, PBR Mushroom Pack Lite, Unity Asset Store, viewed 20 October 2022, <https://assetstore.unity.com/packages/3d/vegetation/pbr-mushroom-pack-lite-56263>.               

Antoine, G 2021, Nature Sounds Pack - Free, Unity Asset Store, viewed 8 October 2022,<https://assetstore.unity.com/packages/audio/sound-fx/nature-sounds-pack-free-20207>.

Arayan, D 2021, ezy-slice, GitHub, viewed 8 October 2022, <https://github.com/DavidArayan/ezy-slice/>.

Brick Project Studio 2022, Apartment Kit | 3D Environments, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/environments/apartment-kit-124055#publisher>.

danielayala123 2020, free banana 3d model, www.turbosquid.com, viewed 23 October 2022, <https://www.turbosquid.com/3d-models/free-banana-3d-model/1071667>.

dinopunch 2018, Living Birds, Unity Asset Store, viewed 20 October 2022, <https://assetstore.unity.com/packages/3d/characters/animals/birds/living-birds-15649>.

Niwashi Games 2019, White Rabbit, Unity Asset Store, viewed October 20 2022, <https://assetstore.unity.com/packages/3d/characters/animals/white-rabbit-138709>.

Pixel Games 2015, ParkChair, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/props/exterior/parkchair-850>.

Pixel Games 2016, Realistic Tree 9 [Rainbow Tree], Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/vegetation/trees/realistic-tree-9-rainbow-tree-54622>.

 reach the enD 2017, Food and Kitchen Props Pack | 3D Props, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/props/food-and-kitchen-props-pack-85050>.

rpgwhitelock 2021, AllSky Free - 10 Sky / Skybox Set, Unity Asset Store, viewed 8 October 2022,<https://assetstore.unity.com/packages/2d/textures-materials/sky/allsky-free-10-sky-skybox-set-146014>.

Schindler, S 2020, free 3ds model orange resolution photorealistic, www.turbosquid.com, viewed 16 January 2022, <https://www.turbosquid.com/3d-models/free-3ds-model-orange-resolution-photorealistic/756550>.

VIS Games 2022, VIS - PBR Grass Textures, Unity Asset Store, viewed October 8 2022, <https://assetstore.unity.com/packages/2d/textures-materials/floors/vis-pbr-grass-textures-198071>. 

Scripting References & Tutorials

Dapper Dino, 2018, How to Add Voice Recognition to Your Game - Unity Tutorial, online video, YouTube, viewed 9 October, 2022 <https://www.youtube.com/watch?v=29vyEOgsW8s&t=757s>. 

LandVR 2020, How to Slice Objects in your Oculus Quest/VR App using Unity, online video, YouTube, viewed 8 October 2022, <https://www.youtube.com/watch?v=iRW2CyQysdw>.

Li, X 2020, Create a Canvas Pointer in Unity for Oculus, online video, YouTube, viewed 8 October 2022, <https://www.youtube.com/watch?v=8fT478uopco>.

Unity 2022, ScriptingAPI: KeywordRecognizer, Unity, viewed 8 October 2022,<https://docs.unity3d.com/ScriptReference/Windows.Speech.KeywordRecognizer.html>.

Valem Tutorials, Two Hand manipulation and Restricted Grab - Oculus Interaction SDK - PART 2, online video, YouTube, viewed 25 October 2022, <https://www.youtube.com/watch?v=P0rbFHWpKnA&t=350s>. 

Valem Tutorials,  User Interface and curved Canvas - Oculus Interaction SDK - PART 6, online video, YouTube, viewed 25 October 2022, <https://www.youtube.com/watch?v=XOc71-Og0Kg&t=141s>. 

Appendix I - Link to Usability Testing Survey

https://forms.gle/KdCcUupmBeMxhVJj7