Augmented Reality: Getting under the skin

Pokémon Go pushed Augmented Reality, or AR, into the mainstream, sending 500 million people around the world chasing cartoon characters on their phones. But now, in a unique multi-disciplinary collaboration, it’s making the leap from entertainment to education.

A new fusion of augmented reality, gaming technology, and anatomy is giving physiotherapy students at the University of Melbourne access to cutting-edge technology to take a look inside the human body by projecting different layers of muscles and bones over the top of a volunteer ‘patient’. It provides an inside view of how the body works as it moves in real time.

The technology, called the Augmented Studio, is designed to radically enhance the teaching of physiotherapy where students currently use their knowledge of anatomy to understand how muscles work beneath the skin of a patient because they can’t see through them. But the Augmented Studio, developed by researchers at the University of Melbourne, bridges the gap between that theory and practice.

By using tracking sensors mounted on a scaffold it projects images of our muscles and skeleton directly onto a volunteer. The images automatically follow the shape and movement of the body, giving students in the studio space an interactive all-round view of how our bodies work. It can even allow them and their teachers to “draw” on the projected image to make information and action more explicit.

“What we are doing is overlaying virtual models of what we look like underneath our skin and synchronising that with real human action,” says Dr Thuong Hoang, who is a Research Fellow at the Microsoft Research Centre for Social Natural Users Interfaces, at the University of Melbourne.

The Augmented Studio was built by Dr Hoang, computer engineer Zaher Joukhadar and Phd student Martin Reinoso, who adapted Microsoft’s Kinect body sensing and tracking device as well as “RoomAlive” projection technology; both of which were originally designed for computer gaming. Once a person steps into the projection space and forms a T-shape with their arms outstretched, the trackers lock on to them and the projected image conforms to their shape and movement.

At the moment the projected ‘overlay’ doesn’t show how our muscles actually move when we contract and relax our muscles. Instead, it tracks the body and movement at the joints. But eventually Dr Hoang wants to add in ‘animation’ that can show the actual movement of muscles as the model moves.

University of Melbourne physiotherapy lecturer Dr David Kelly says the students quickly embraced the technology during pilot sessions in 2016, which are continuing in 2017. He says the combination of live movement and interaction, in which students could actually move and feel the model’s limbs, helps them to grasp the relationship between their learned anatomy and how it works dynamically.

“For first year students it can be really hard to bring together anatomical knowledge with how the body actually works because it can be difficult to visualise. But when they see a real person who they can interact with, while also seeing the muscles and skeleton projected over the top, combined with the ability to draw and write on the body, it all becomes much easier for the students to learn about how the body moves,” says Dr Kelly, from the Melbourne School of Health Sciences.

The Augmented Studio also provides a more visual and intuitive way of learning that Dr Kelly says will benefit those students who naturally learn more easily by direct visualisation, rather than through reading and listening. “There has always been a group of students that struggle because the limited ways in which we have to teach may not conform to how they learn best,” he says.

Developments in AR, which seeks to use technology to enhance what we can already see, hear and feel in the real world, are far ahead of chasing GPS tracked Pokémon. There are viewing devices such as glasses that can overlay what we see with three-dimensional graphics, video and holograms, and we can generate projections like games that people manipulate by moving our hands.

The big advantage of the Augmented Studio over advances like 3D holograms is that the students can actually touch and move the body, making it a much more interactive experience. They also don’t have to wear headgear, which means it could potentially be used in bigger settings with larger numbers of students.

“It has always been hard to capture the dynamic side of how our anatomy works, so the difference here is the high level of interaction you can achieve. The student can, for example, ask the model to kick and they can then look at variations from different angles at what is happening as someone kicks,” Dr Kelly says.

The Augmented Studio is still in early-stage development and Dr Kelly would love to see it migrate to using muscle animations. Dr Hoang is also working to develop a system for the student interaction with the model to be automatically recorded onto their tablets so they can have a permanent record of what they were learning.

Another challenge is to find a way to make the studio more transportable and quicker to set up. At the moment the studio can work very effectively in a dedicated tutorial space where it could be permanently set up, but Dr Kelly says a more portable set up would increase its flexibility for teaching.

The Augmented Studio is an extension of Dr Hoang’s earlier work exploring how Virtual Reality and body tracking could be used to help guide body movement for dance and marital arts students. Arising from a collaboration between the Physiotherapy department’s Teaching and Learning director, Associate Professor Louisa Remedios and Professor Frank Vetere, Director of Microsoft Social NUI, Dr Hoang started working with the physiotherapy department on developing a teaching aid. He then realised that Virtual Reality, in which you are immersed in an entirely created world, wasn’t suited to teaching physiotherapy that is very hands on.

“When we got into the class rooms we had to change our thinking. VR just wouldn’t work in the tactile environment in which they learn and practice,” Dr Hoang says. It was when he noticed that students kept referring back to anatomy charts when they were practicing on each other that he started thinking of using Augmented Reality to put the virtual muscles on the body

Dr Hoang is now working on extending the tracked projection technology to various health and fitness areas, and even in performance art. He says that using tracking sensors with projections it is possible to create guides that show people how to position their bodies for practising fitness, sport and dance.

Using virtual reality headsets he and PhD student Martin Reinoso have already developed a prototype that allows a martial arts teacher to remotely instruct students on the right position to hold. By using body tracking and linked headsets student can match their movement to align with those of their teacher. There is also scope to project information on our own body’s performance, such as heart rate and breathing, so it is visible either on our projected selves or on a nearby surface.

“The innovation we have created isn’t just limited to the fixed information that we have been projecting so far. If can be used to project dynamic information onto yourself or any surface around you,” Dr Hoang says. “All of what I’m dreaming of is very possible.”

Dr Hoang and his team will be running a demonstration of the Augmented Studio system at Melbourne Knowledge Week as part of the Digital Studio at the University of Melbourne on May 2.

Banner image: Supplied.