Assignment 4 Report

I.                 Introduction

In-Progress Screen Capture:

Pitch

Do you have trouble maintaining everyday physical motions such as opening a door, balancing objects, using tools? Or perhaps you struggle with some cognitive and mental functions such as reaction time or memory? This VR application is designed to help the elderly identify, measure and improve their everyday motor skills, reflexes, and mental/cognitive functions!

II.                 Description of the application 

This VR application is designed to help elderly people maintain physical and cognitive functions by providing simple tests of “real-life” scenarios in realistic indoor and outdoor settings, and collecting performance data to display to a caretaker afterwards.  The aim of physical tests is to create scenarios, such as opening doors and picking up objects, that provide an opportunity to safely use and maintain motor skills that have decreased with age. A person’s mental capacity will also decrease with age, so this side will be tested and trained through memory and reaction challenges.

According to the World Health Organisation (2022), the proportion of elderly people in the population is increasing worldwide, and therefore it is vital for health systems to be prepared for this shift. Physical and social environments play a large role in healthy ageing, so it is important to remove obstacles to being physically active and healthy to reduce the risk of non-communicable diseases. Our application aims to make it easier to create an ideal environment for the elderly by offering a novel experience that engages them, and also assisting caregivers by helping to automate some activities and collect data relevant to health.

This is especially important in Australia, where the elderly care business has long been plagued by issues as found by the Royal Commission into Aged Care Quality and Safety (2021). According to the commission's recommendations, "more investment in pre-certified assistive technologies and smart technology is needed to support care and functional requirements of older people, help manage their safety, and contribute to their quality of life” (2021, p. 149). Therefore, an application that provides opportunities to practice and assess physical and cognitive skills could be beneficial to the aged care industry.

III.              Description of the interface solution 

Various interfaces can be used to perform physical and cognitive tests, such as PC platforms, AR, or VR. On PC there are many games, websites, and video tutorials available to everyone. However, the elderly may be unable to sit for extended periods of time in front of a computer screen, and using a computer mouse for input creates limited interactions. Users can interact with various elements of the AR interface, such as buttons and cameras but this does not produce “real” interactions; for example users cannot use their hands to grip or catch things in the game. Because of the limitations of phone and iPad screens, people might find it challenging to interact with cognitive test elements. 

As a result, developing applications utilising VR interface technology was considered the best option. When compared to a typical 2D desktop interface, VR programs can detect a wider range of input e.g. whole-body gestures, eye-tracking and 3D interactions.  Users may engage with VR and complete activities in a pleasant and realistic manner - sitting, lying down, or being in whatever position is comfortable for them.

Virtual technology can be used to deliver both physical and mental benefits for older persons. Compared to a traditional interface, using VR controllers allows elderly people to engage their upper torso, neck, core and arms which can improve their physical health (Maplewood Senior Living 2021). Virtual interfaces can also better engage the brain compared to traditional interfaces with a study showing an 8.8% improvement in memory recall for VR users (Krokos, Plaisant & Varshney 2019), making VR ideal for improving cognitive ability.

IV.              Interaction Design

Handling impairments, informing and educating caretakers, developing trust and security, and introducing virtual worlds carefully are all things that should be considered when constructing an enjoyable and safe VR experience for the elderly (Bohlin & Müller 2020, p. 47). Considering accessibility first and foremost, we decided to implement voice control in the application. A study of different user interfaces in VR showed the speech control is better for ease of learning and uncomplicated handling compared to 2D and 3D UI (Weiß et al 2018, p. 57), making it ideal for this context. The user can say keywords such as “kitchen” to move to the desired location, however the transition may be too sudden given the need to introduce virtual environments slowly. In future we hope to address this, as well as having the application recognise numbers, which will be used to answer questions in a memory test. 

In the indoor kitchen scene, the controllers are used to interact with grabbable objects, allowing the user to pick up a knife or apple, and also use the knife to slice the apple into pieces. In the future, it is intended for the apple slices to be grabbable so that they can be arranged on a plate, and for various other cooking supplies to be located in the kitchen which can be prepared to make a meal. This scenario will be used to assess physical coordination in a simulated everyday situation that poses less risk to the user, as well as testing cognitive ability to follow some recipe instructions.

In the outdoor park scene, we will develop an interaction where the user moves the controllers and reaches out to touch various surrounding objects. This exercise will help the user build gross motor skills and will collect data on their flexibility. On the cognitive side, we intend to implement a memory test where the user observes a scene, and then afterwards uses their voice to answer a question i.e. “how many birds were there?” We would also like to measure the reaction response of the user, but are still determining an appropriate scenario without presenting stimuli in a shocking or sudden way.

While the user performs tests, various performance data will be collected, and at the end of the testing session a summary will be displayed. This snapshot is intended to provide a general indication of the user’s strengths and weaknesses and highlight any areas where they did not perform as well as expected. A caretaker can then use this information as needed to better address the needs of the elderly user.

V.                Initial technical Development

This application will be developed using the Oculus Quest VR headset and touch controllers and built in the Unity engine.  It will contain three scenes: a main menu, a kitchen and a park, and a basic version of each has been developed.

A main menu to access the different scenes was implemented and can be interacted with using either a raycast from the user’s hand, or through voice commands.

In the kitchen, the ability to pick up a knife and cut an apple was implemented by following a video tutorial (LandVR 2020) and using open-source mesh slicing code from GitHub (Arayan 2021). However, some further work will be done to solve visual and physics-related bugs that occur during this interaction. In future, the user will also be able to open the fridge and grab various other cooking supplies to follow a recipe.

The proposed mobility and memory tests for the outdoor scene will be developed in the future. A generic reaction test has programmed where the reaction signal is fully customisable, as the script calls Unity events to set up the scenario. When the user presses the trigger, their reaction time will be saved in milliseconds.  We will continue to consider the most appropriate way of testing reaction time for an older target audience, and create new game objects to be used in the test accordingly.

The voice recognition was implemented through the Unity Windows library “using UnityEngine.Windows.Speech'' that uses the Cognitive Speech Services SDK created by Microsoft. The voice recognition in the main menu is designed for eldery users who may not have the ability to navigate the menu with VR controllers. With voice commands such as ‘menu’, ‘brain’ and ‘body’, the user can switch between scenes, and the number of recognised words will increase in future development.

VI.              Initial 3D Models 

Fridge  (Brick Project Studio 2022) - Grabbable item - the user can open three doors of the fridge, and will be able to take out food supplies in future.

Apple and knife (reach the end 2017) - Grabbable items - Everyday items found in a kitchen that can be used to simulate a basic interaction of cutting up an object.

A realistic tree model by Pixel Games (2016) is used to decorate the outdoor scene, making it more immersive.

A bench model by Pixel Games (2015) is also used to make the scene feel more like a real park.

VII. Conclusion 

To summarise, our application is designed to provide simple physical and cognitive tests for the elderly in realistic indoor and outdoor virtual settings.  We currently have basic interactions such as cutting fruit in the kitchen, measuring reaction time in the park, and using voice control, and in future development we will fully implement these existing tests as well as adding new ones to measure memory and range of motion.  We will also record more data to be displayed after testing sessions for the benefit of caregivers.

VIII.           References

Report

 Bohlin, E & Müller, L 2020, Designing Virtual Reality Experiences for Elderly Degree Project, Linnaes University Sweden, viewed 8 October 2022, <https://www.diva-portal.org/smash/get/diva2:1445128/FULLTEXT01.pdf>.

Krokos, E, Plaisant, C & Varshney, A 2019, ‘Virtual memory palaces: immersion aids recall’, Virtual Reality, vol. 23 no. 1, pp. 1-15.

Maplewood Senior Living 2021, Benefits of Virtual Reality for Seniors, Maplewood Senior Living, viewed 30 May 2022, <https://www.maplewoodseniorliving.com/blog/benefits-of-virtual-reality-for-older-adults/>.

Royal Commission into Aged Care Quality and Safety 2021, Final Report: Care, Dignity and Respect, vol 1.

Weiß, Y, Hepperle, D, Sieß, A & Wölfel, M 2018, ‘What User Interface to Use for Virtual Reality? 2D, 3D or Speech – A User Study’, 2018 International Conference on Cyberworlds, pp. 50-57. 

World Health Organisation 2022, Ageing and health, World Health Organisation, viewed 8 October 2022, <https://www.who.int/news-room/fact-sheets/detail/ageing-and-health>.

Assets          

Antoine, G 2021, Nature Sounds Pack - Free, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/audio/sound-fx/nature-sounds-pack-free-202076>.

Arayan, D 2021, ezy-slice, GitHub, viewed 8 October 2022, <https://github.com/DavidArayan/ezy-slice/>.

Brick Project Studio 2022, Apartment Kit | 3D Environments, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/environments/apartment-kit-124055#publisher>.

Pixel Games 2015, ParkChair, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/props/exterior/parkchair-850>.

Pixel Games 2016, Realistic Tree 9 [Rainbow Tree], Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/vegetation/trees/realistic-tree-9-rainbow-tree-54622>.

 reach the enD 2017, Food and Kitchen Props Pack | 3D Props, Unity Asset Store, viewed 8 October 2022, <https://assetstore.unity.com/packages/3d/props/food-and-kitchen-props-pack-85050>.

rpgwhitelock 2021, AllSky Free - 10 Sky / Skybox Set, Unity Asset Store, viewed 8 October 2022,<https://assetstore.unity.com/packages/2d/textures-materials/sky/allsky-free-10-sky-skybox-set-146014>.

VIS Games 2022, VIS - PBR Grass Textures, Unity Asset Store, viewed October 8 2022, <https://assetstore.unity.com/packages/2d/textures-materials/floors/vis-pbr-grass-textures-198071>. 

Scripting References & Tutorials

Dapper Dino, 2018, How to Add Voice Recognition to Your Game - Unity Tutorial, online video, YouTube, viewed 9 October, 2022 <https://www.youtube.com/watch?v=29vyEOgsW8s&t=757s> 

LandVR 2020, How to Slice Objects in your Oculus Quest/VR App using Unity, online video, YouTube, viewed 8 October 2022, <https://www.youtube.com/watch?v=iRW2CyQysdw>.

Li, X 2020, Create a Canvas Pointer in Unity for Oculus, online video, YouTube, viewed 8 October 2022, <https://www.youtube.com/watch?v=8fT478uopco>.

Unity 2022, ScriptingAPI: KeywordRecognizer, Unity, viewed 8 October 2022, <https://docs.unity3d.com/ScriptReference/Windows.Speech.KeywordRecognizer.html>.