The role of sound in Virtual Reality (VR) is a factor that can oftentimes be overlooked by the user because of the novelty of the visual experience, however, as people become more accustomed to and expectant of the technology, immersive audio may very well dictate who reigns supreme.
Firstly, it is important to note that ‘hearing’ is the fastest of our senses.
Audio recognition happens after just 125 milliseconds compared to the 500 milliseconds required for visual stimuli to be perceived. So what does this mean? Quite simply, bad audio runs the risk of diminishing the immersive experience – and what’s VR without optimized immersion?
During the MavricVR event in Cork City, Ireland, I asked Robert Scoble – entrepreneur in residence at Upload VR – to talk about the role of sound in VR and the difficulties in recreating real-world ambient noises. Read about Cork City’s first VR conference here.
Natural ambient sounds, directional audio, head-tracking and advanced ambisonic recording techniques will all be required for sound to come to life – and for the VR experience to feel real – but until then, many will continue to incorporate music into their products.
Of course there are VR experiences that intentionally incorporate music as to illicit emotion and to draw on one’s attention, but sometimes, the music can distract, as it’s telling you what you ‘should’ be feeling – or at the very least tries to massage you to a particular point – as opposed to letting you explore your feelings yourself.
We don’t have music in our playing in our head in real life, so should we in experiences that claim to replicate it? Using music to further immerse the viewer is common practice in the movie and TV industry, so why not VR? With music, you’re moving into the domain of ‘entertainment’ as opposed to trying to recreate a real world simulation.
Speaking at the Virtual Reality World Congress (VRWC) in Bristol this month, Thomas Bible of Submersion Audio talked about the difference between informational and emotional audio – and the functions associated with the two. As you can see in the image below, whether 2D or 3D, music is heavily emotionally orientated.
I wondered what the benefits and challenges were with incorporating music into your VR experience, but of course that depends on what you’re trying to achieve.
Immersive VR Education‘s Apollo 11 is a perfect example of how VR developers can use music to enhance the user-experience. Apollo 11 VR brings you on an adventure through time and space, in a highly emotive, entertaining, information-laden experience. As stated by Immersive VR Education CEO Dave Whelan, “We’re making these for children so that they’ll learn history through experiences like this instead of just reading it from text-books”.
It begins when you are placed in a room to watch a rousing speech by JFK and from here you commence a tour of the many different missions. Whether the launch-pad or cockpit, the moon orbit or landing, the experiences are accompanied by theatrical music that can only be described as EPIC.
The music is especially effective in the passive version of the experience – where you don’t have to control the lunar module landing or the other more interactive experiences – as the blend of music, narrative and visual stimuli, all combine for a deeply immersive out-of-this-world emotive experience.
And that’s the thing about humans, we tend to remember emotionally charged events.
As a learning tool, I think this approach to teaching could be immensely beneficial to students who oftentimes struggle with the cyclical routine of the school curriculum. Why not leave the class for a half hour to go to space? Especially if you’re going to learn something. But this is where we might consider the relationship between attention, memory and emotion.
Are you learning?
It seems a silly question, and the obvious answer might be yes, but if this experience wants to be utilized for ‘educational’ purposes, I forsee it will require empirical evidence to back up its claims in order to be accepted by established educational institutes. To show that it can do just as good a job – if not better.
Immersive VR Education have this ‘Cone of Learning’ table on their website which provides a nice visual representation of learning-styles, however, more detailed statistical insight is required in order to assess experience, engagement and memory-recall.
One study published the Journal of Scientific Research & Reports showed that ‘happy’ and ‘sad’ mood priming can have an impact on memory recall. Under this assumption, it would be extremely important to measure the information recall – particularly with emotional reactions like this. Is there a danger that intense or alien visual stimuli takes away from the more informative aspects of the experience – and is the same the case for the music?
Or is it the whole package that makes it work?
The next step for Immersive VR Education is to validate their VR experience, as to ensure it is in fact educational, and not disruptive or distracting – both inside and outside (after) the experience – across demographics, as to determine how different people interact with the application. It is very easy for me to say that I enjoyed the Apollo 11 Experience and took a lot away from it, but I’m more used to slipping on the VR headset than most.
Some find it exciting, and others nauseating. It may come down to individual differences but these are questions that need to be asked before claims of any positive transfers of learning. Pre and post measures with multiple control groups as well as eye-tracking technology might be a good place to start – to determine recall as well as highlight any notable distractions in the VR environment.
Although these considerations come down to the design of each individual experience – it is imperative that the contribution of the learning tool be quantified statistically and empirically.
The potential for using VR with children and students is something to be encouraged at one level, but we also need to be concerned about the risks. What I’d like to see is a controlled study where the same information is conveyed through many different teaching environments with suitable measures deployed. The Apollo 11 VR experience is is definitely entertaining, and definitely educational – but the question is, to what extent?