HKU explores immersive virtual experiences in education

Sound as compass

During the experiment, a team of lecturers from both academies developed a VR experience. In this experience, a player wearing a VR headset moves through a series of virtual landscapes, from a dark forest to a dramatic plain and a cathedral, while navigating by sound alone. In this way, sound acts as a kind of compass.

Between physical and virtual space

In the experience, various techniques and systems from both academies converged. The landscapes were created by Games; the virtual acoustics were realised via a multi-channel audio system developed by M&T. The system has wooden speaker boxes that can simulate different acoustic environments. The player’s voice and sounds of movements were captured in real time and replayed with diverse reverberation and echo profiles.

The player moves physically through the studio, while lights, sound and projections react to the virtual environment. A portable speaker plays the role of a virtual flashlight, interactive object, and physical sound source. Altogether this creates a hybrid “playing field” in which the boundaries between physical and virtual space are continuously rewritten. And because of the virtual acoustics, the player becomes even more immersed in the world.

Multidisciplinary team

A diverse group of HKU participants is involved in the experiment, including lecturers from Games and Music & Technology, enabling a transdisciplinary approach that combines technical, artistic and educational perspectives. The project is led by the XR-team, part of HKU’s Department of Education, Research & Innovation. HKU's professorship (Dis)Connected Technology & Creativity is also closely involved.

From experiment into the curriculum

Prior to the experiment, each participating lecturer formulated their own research question. During the experiment, they tested these questions. In doing so they explored how virtual acoustics can be integrated into game-design processes; how lighting, motion capture and audio together can form an immersive dramaturgical system; how XR can open up new forms of interactive narratives and game rules; and which educational skills and technical infrastructure are required to enable students to work with XR.

'It was a nice first exploration: fruitful and informative'

Antal van Nie, the Music & Technology teacher who developed the audio system with the wooden speakers for simulating spatial sound, wanted to find out what kind of infrastructure is required to create immersive experiences with virtual acoustics in game environments. 'That question has now essentially been answered, but the details aren’t in yet. It was a promising first exploration: fruitful and informative. I’m keen to continue developing the right infrastructure, both software and hardware, and to design workshops for students and workshop supervisors, so that we can then integrate these into the curriculum on an annual basis.'

'My idea was that the shared character of the vistual acoustics could lead to a VR experience that would feel less isolating'

Games-teacher Aaron Oostdijk entered the experiment with the question of how, by focusing on sound and acoustics, you can create a meaningful experience with game technology. ‘My idea was mainly that the shared character of the virtual acoustics could lead to a VR experience that would feel less isolating, and therefore perhaps easier for non-VR players or audiences to experience alongside the VR user. That turned out to be partly true, though the visual component remained important. If we want to use the demo we created as an example in our teaching, we must first need to simplify and standardise it a little further – and then adapt it into a format that would allow students to work with it, for instance with support from Blackbox staff.’

Implementing IX / XR across education

In the next academic year, more experiments with immersive technology are planned at HKU Theatre and Media. The next step is for the XR-team from the Department of Education, Research & Innovation to join with the course leaders of various academies and submit a proposal for gradual implementation of IX and XR in teaching. The XR-team’s project leader, Chantal Fortes, said: ‘XR doesn’t just change our tools, but also our way of seeing, listening and making sense. Through collaborative experimentation we discover new forms of art, education and cooperation.’