SIMQ: Virtual and Augmented Reality

From the Desk of Carol C. Dudding, PhD, CCC-SLP, CHSE

In this article we will explore the opportunities afforded by emerging technologies in virtual and augmented reality. You will be provided information about the current state of these immersive technologies and challenged to consider their potential in clinical education. This article will provide a mix of theory and practice. We will present information with the intent to stimulate ideas.  

 

This article is part of an ongoing series of articles on simulations in healthcare, specifically in Communication Sciences and Disorders (CSD). The SimQ format, allows us to explore the technological as well pedagogical aspects of simulation for clinical education through the words of the experts.  It is anticipated that the audience will include university faculty, clinical educators, and professionals with interest in the use of simulations to enhance the clinical education of our students and practicing professionals. These articles are intended to address the needs of those considering the use of simulations, and those with varying experiences and level of skill.

 

Carol C Dudding, PhD, CCC-SLP, CHSE
Contributing Editor

SIMQ: Virtual and Augmented Reality

In this installment of SimQ, I am interviewing our contributing editor, Carol Dudding, about the emerging technologies in virtual and augmented reality.  She is currently involved in some projects that I believe you will find interesting and more importantly, thought-provoking. Carol, I’m looking forward to learning more about this topic and how it might apply to simulations.  

I am glad to be able to share what I know about VR/AR and mixed realities. It is an area that is exploding in the commercial marketplace - especially advertisement and gaming industries. There is also a lot of work being done in regards to military training. I am very excited about the application to clinical education. I think it could be a game changer for simulations. I believe it will provide an avenue for our students to have access to simulated experiences directly related to our scope of practice.  

Q.  What do you mean by AR/VR?

Given the emerging nature of this technology, this may be one of those times when it is appropriate to refer to Wikipedia …“Virtual reality (VR) is an interactive computer-generated experience taking place within a simulated environment. It incorporates mainly auditory and visual feedback, but may also allow other types of sensory feedback like haptic.” https://en.wikipedia.org/wiki/Virtual_reality. VR is an umbrella term that encompasses a group of technologies that allow for the user to experience a simulated environment. These technologies vary in level of immersion and interactivity. Many VR experiences take place through some sort of viewing device, like Google Cardboard, Oculus Rift or HTC Vive. These devices allow for an enhanced visual experience. Everything looks 3D and the perspective changes as you look around the environment.  In some cases, the interaction takes place on a computer monitor in the form of a 3-D computer-generated persona, known as an avatar. SecondLife is an example of such a technology. Although SecondLife is seen as obsolete in comparison to the newer interfaces. To take it to the extreme level, there are fully immersive experiences that take place in specialized rooms.  Interaction within VR takes place with use of game controllers, motion detectors, haptic gloves or even hand motion. You can incorporate sound, touch and even smells through peripheral technologies. If done well, it's an amazing experience.

Q.  Sounds intense. So what is AR and how is it different from VR?

Augmented Reality (AR) is a form of VR. Let's refer back to Wikipedia for a definition and then we can unpack it a bit.  “Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The overlaid sensory information... is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment.” https://en.wikipedia.org/wiki/Augmented_reality.  Perhaps the best example is the Pokemon-GO phenomenon. If you recall, in the summer of 2016 millions of people were walking around staring at their smartphones looking for Pokemon - small fictional characters “hidden” in their favorite park or coffee shop. This is an example of augmented reality. As the name implies, it augments the real world.  It overlays virtual aspects onto the real world.

Q.  My kids were all about Pokemon-GO. Pokemon-GO is a game, and I am very aware that these technologies are transforming the gaming world. Do you have another example of AR? 

Sure, recently I was shopping online for a new ceiling fan for my home. I was able to take a photo of my room and then virtually insert the new ceiling fan into the image, allowing me to see what the fan would look like in my home. That’s another example of augmented reality for the retail industry.  

There is yet another classification of technologies, a hybrid between AR and VR, known as Mixed Reality.

Q.  I think I can guess what that entails based on the name “mixed reality”. Tell us more.

To be honest I have a hard time truly grasping and explaining the concept of mixed reality (MR).  It is described as anchoring objects in the real world within the virtual world. – Pokemon-GO was an overlay but mixed reality puts real objects within the virtual experience. MR combines computer processing, human input, and environmental input to create a highly immersive and interactive experience. An example is Microsoft Hololens which allows the user to view virtual images placed in the physical world while wearing a headset. In medicine, researchers are using mixed reality to allow learners to interact with a physical manikin while manipulating organs within in a virtual environment. My current project is a form on mixed reality. More on that later. Just imagine the possibilities if all students had access to a virtual cadaver as part of their anatomy and physiology course. For more information search the internet using the terms “medical mixed reality”.  

Q.  Are there any other forms of VR?

The types of VR range in levels of immersion and the extent that the technology blends with the physical world. There doesn't seem to be total agreement on the definitions at this time. A partial list of the platforms includes web-based gaming, 360 videos, fully immersive VR rooms, wearable goggles, holograms, and treadmills. All these peripherals are intended to give the user a fully interactive experience.  For example, a CAVE experience is a room or building that incorporates sights, sounds, sensation and smells designed to simulate a real-world experience. A company known as The VOID is opening up a series of storefronts using state of the art immersive VR technologies. The technology allows you to become part of the Ghostbuster movie and hunt ghosts. The Void describes the experiences as “whole body, hyper-reality”.  We are hoping to take our VR/AR students on a field trip to one of these places in the spring.

Q.  I see why you are excited about this class of technologies. How does anyone keep up with it all?

Carol: I am amazed at how rapidly the technologies are advancing. Just a little over 2 years ago I purchased my first 360-degree video camera. Now the technologies are so far advanced, it makes my 360 camera seem old and obsolete. The same can be said for the programming software and peripherals that allow for programming and interactives. It's mind-blowing but exciting.

Q.  Back to reality (pun intended).  What does all of this have to do with clinical education and more specifically, simulations?

If I may, I’d like to take you back about 15 years and share with you an experience I had while a clinical faculty member. I was teaching and supervising in a Communication Sciences and Disorders (CSD) graduate program. I had sat through a number of oral comprehensive exams and found myself wondering, “what are they (students) thinking?” And the truth was, none of the faculty knew for sure how or why the students arrived at their answers.  We could only infer from the students earlier written response and the reasoning put forth in the oral defense. It began a long-standing search in my mind, for more authentic ways of teaching and assessing skill performance in our graduate programs.

I will fast forward to the last five years when I became interested in simulations as a form of clinical education. I was a team member on a large federal grant, part of which involved the use of simulation technologies for the education of healthcare students including speech-language pathologists, nursing and mental health professionals.  As part of this collaboration, I had access to the simulation center on my campus and was spellbound by the high tech manikins employed for the training of nursing students. I was really knocked off my feet when I realized the resources (financial, personnel and space) required to run even a modest simulation lab. I quickly understood why programs in CSD don't have their own sim labs.  

And that brings me up to date and my current interest in VR technologies. I see VR technologies as a means of making the benefits of simulations applicable and accessible to our professions. That’s not to say that CSD programs don't have access to high tech manikins, standardized patients or other simulation technologies. In fact, more and more programs are utilizing simulations for clinical education. But it is most likely that the sim labs are housed in nursing and/or medical programs.

Q.  Sounds promising. What equipment do I need to purchase and how much does it cost?

Carol: Katie, you know me well enough that I’m not gonna answer that question without first reminding everyone that simulations, like other technologies, are a tool for learning. In order to be effective in teaching and learning, these tools must be embedded according to  best-practices in clinical education. It all starts with the learning objectives. It is always a mistake to lead with a technology as the focus.

Q.  I was setting you up with that last question. Based on my experiences with Simucase technologies, I couldn’t agree more. The pedagogy of simulation is a huge topic that we have covered in part in other articles. For those interested in simulations of any form, we strongly recommend that you seek out that information. With that said, In terms of using VR technologies for educational purposes, where can we look for information?

As you might expect, the available information on new and emerging technologies such as VR is, well, new and emerging. The amount of information on VR for educational purposes is even more limited. There is a website, https://www.virtualiteach.com/, keeps me informed about what others are doing in terms of integrating VR into education. I suggest you check it out for more information.

Steve Bambury hosts that site. He has created a terrific graphic that represents VR along a continuum of interactivity ranging from perception, stimulation, interaction to a level of immersion. Perception is the lowest level of immersion. Using a 360-degree video is an example of perception and is the most accessible form of VR technology. The next level is simulation - pairing audio with a 360-degree video experience. Programming software, hotspots, peripherals and controllers add a level of interaction necessary for an effective learning experience. The fourth level, immersion, involves multiple senses and allows for full interaction within the environment. Readers are encouraged to consider each of these levels, and select the form of technology needed based on the identified learning objectives,

Q.  What do you see as the advantages of VR for clinical education?

Very few CSD programs have full access to a manikin-based simulation center. This can be due to a number of factors including institutional policy, costs in terms of personnel and equipment, and frankly, a lack of knowledge on the part of CSD faculty. For those programs able to navigate the barriers to access to simulation centers, there remains that fact that most high-tech mannequins were not designed to offer our students experiences specific to our scope of practice. So we are often modifying our simulations and hoping to approximate experiences. Well, I imagine that VR can meet some of those challenges. First, the VR experiences would require significantly less equipment, cost, personnel and space as compared to a sim lab. The VR experiences could be designed with specific clinical skills relating to speech-language pathology and/or audiology.

Q.  What do you see as other potential uses for VR technologies in CSD?

I am also excited about the application of VR technologies to create a form of standardized, authentic assessment for our students. Other healthcare professions have a long established use of OSCEs (objective structured clinical examinations) to assess knowledge and skills. As the name implies, the benefit of an OSCE is that it is objective and structured. It allows for authentic evaluation of specific clinical and professional skills. I believe simulation technologies, specifically VR technologies, hold great promise in this area.

Q.  Are you aware of any virtual simulations exclusively for our audience?

My response is tempered by what is meant by virtual simulations. But currently I am not aware of any simulations specifically for speech-language pathology/audiology using virtual reality. There are a number of them available for nursing and medical education. [audible sigh]. There are a few gaming applications that could be adapted and used for our purposes. You can get a free account to SteamVR and search existing games that may suit your needs. Keep in mind, this is a platform for gamers but there is a lot of really cool stuff on that site.  

Q.  Can you tell us about your work in this area?

I am working with a very talented group of colleagues and students in designing a mixed reality simulation to allow students to practice and experience a swallowing exam. We are using the procedure known as FEES (fiberoptic endoscopic evaluation of swallowing.) This is a highly invasive procedure with considerable risk to the patients. A select number of students in our program are able to observe a FEES procedure but none of them have the actual experience while students. This simulation is designed to give the students the virtual experience of conducting and interpreting a FEES examination. This would assist students in bridging the gap between academic learning and skill development, helping to prepare for future clinical experiences.

Q.  So how does it work?

It is still in the development phase but it allows the student to practice placing a model of a fiberoptic endoscope (created with a 3D printer) into the nasal passage of a Styrofoam head. All the while the student will be wearing a VR headset that allows them to interact with an avatar of a patient in a simulated clinical examination room. The model of the endoscope and the Styrofoam head are the actual objects being integrated into the virtual world - hence mixed reality!  The student will see an actual video from a FEES exam, and be asked to correctly label anatomical parts of the upper airway as they virtual pass through the upper airway. Once they reach the level of the vocal folds, the student prepares to begin the examination of the swallow mechanism. The student begins by deciding which consistencies should be administered to the patient, and in what order. The student will interpret the function and safety of the swallow based on the video presented.  The student will receive a score at the end of the experience.

Q.  Wow, that sounds like a great learning experience.

My team and I are incredibly excited about the possibilities. It is our hope to secure funding to create additional simulations using this advancing technology. By the time this article is published, we will have piloted it with graduate students.

Q.  So what do you see as limitations for this type of simulation?

I think many of the limitations stem from the newness of the technologies. The technologies are emerging so rapidly that there is a chance that the technology used in a project become obsolete in a relatively short time. It is difficult to find the necessary talent necessary to build these simulations. Not many are familiar with the programming and technical skills required. As described in an earlier question, some VR technologies are limited in the level of engagement and emersion. The higher the level of interactivity and immersion, the greater the technical resources required. Along with that is the limited transferability of the technology platforms. Currently, there are only a few companies manufacturing the headsets and stations used in VR. These platforms are not always compatible. I am looking forward to advances to allow multi-user interfaces to allow for interactions among learners.  Perhaps the biggest limitation in my mind is the challenge of applying these technologies to what we do as clinical educators and/or clinicians. I’d love to know what the readers are thinking in terms of application.

Q.  Who are you working with on your project?

I am fortunate to be working with Dr. Erin Kamarunas. She is my content matter expert in all things dysphagia. James Barnes is an adjunct lecturer in our X-Lab. It’s a center within our university that is dedicated to fostering new technologies. Dr. Kevin Phaup is with our Industrial Design program. He brings expertise in the user interface, design and creates all the wonderful “props” we need for mixed reality. It truly is an “A+” team. We also have a number of students working with us, some from computer science and some from Kevin’s program.

Q.  How can we find out more about these technologies?

I have found a number of free webinars on the topic. Most of them are focused on commercial and retail interests but it's a good place to start. Once you sign up for one of those and your contact info is out there, you will begin to see lots of things appear in your inbox. For example, VeeR VR is a community of those interested in VR content delivery. As mentioned earlier, Virtualiteach is focused on VR for education. There is an organization called ARVRA that provides members with newsletters, and a venue for networking and dissemination of research.

Q.  What do you see as the future of VR/AR? (collaborative)

I am expecting that we will see VR technologies run the course of other technologies. As it becomes more prevalent in the commercial and retail realms, I think we will find spill over into clinical education. As VR becomes more a part of our everyday experiences, as is predicted by many investors including Facebook, the technologies will likely become more accessible and cost-effective.