HoloBand
We collaborated with audiologists from Rigshospitalet's Centre for Hearing and Balance in Copenhagen; and a heard-of-hearing musician to find out how AR technolgy can help music perception training. We designed an audiovisual spatial experience to train various aspects of music perception. Accompanied by a secondary computer interface, where audiologists can control the spatial interface and create different exercises. Via a focus group testing with 7 audiologists, we found that our solution has potential to be integrated in their clinical workflow.
Spatial design
AR interactions
3D UIs
User testing
01. Challenge
How can we make music perception training more visual, contextual and spatial for hard-of-hearing folks through an AR experience via the HoloLens 2?"
Listening to music is generally not really enjoyable for hard-of-hearing people. Whether they use hearing aids or cochlear implant devices, perceiving musical sounds is challenging for them. Luckily this can be trained with different listening exercises. We focused on these and designed an AR experience where clinicians could perform listening exercises with the spatial and visual aids.
02. Research & Discovery
We worked together with an audiologist and a hard-of-hearing musician to create a more engaging music training experience via the HoloLens.
We followed a participatory design process, actively involving them in the design process. They had interesting duality as target users, providing us inputs from both sides of the same experience.
They both underlined that the current state of music perception is very analogue, dry and doesn't support spatial sound and visuals.
03. Ideation
First we held a scoping workshop, where they got familiar with Augmented Reality by trying out different experiences. By directly experiencing the technology, they could come up with feasible directions and ideas.
We settled for a spatial band, each holographic musician playing their instrument "around" the HoloLens user. This could facilitate the three most common training exercises:
Timbre discrimination
Audio localization
Pitch recognition
04. Design & Prototyping
We came up with a secondary interface for the clinicians. During our second workshop we found out that the clinicians would prefer to have control over different sound parameters and visuals.
Secondary interface for audiologists
Here they can monitor the hard-of-hearing patient's position in-real time. After placing the holographic musicians around their patient from the left sidebar, audiologists can reposition the musicians on the canvas. This changes their corresponding position in the spatial environment. Audiologists can also mute musicians, change their pitch and volume, and filter out specific frequencies.
05. Testing & Results
HoloBand was tested with 7 audiologists in Righshospitalet's Centre for Hearing and Balance. After trying out both interfaces, we held a Focus Group with all the audiologists. We were curious how can our developed solution fit into their workflow.
FOCUS GROUP
"I think there was an interesting example where I asked you what instrument is in your left ear? And it was the guitar. And you couldn’t really tell what it was because all the others were playing on the right. And then I added the visuals, and then you could actually hear the guitar."
Through analyzing the qualitative data via a thematic analysis we uncovered 7 themes on how can our prototype can support clinical auditory training.
07. Impact
We published a research paper about HoloBand at the conference on Audio for Virtual and Augmented Reality in Redmond, Washington in 2022.
HoloBand was also one of the first successful projects in an ongoing collaboration between Rigshospitalet and Aalborg University Copenhagen.
2023 — Made with Framer, Spline, React and Three.js. Feel free to reuse any thoughts or ideas.
13:36
I'm based in Copenhagen, Denmark
I'm a designer, researcher, code tinkerer, accessibility tech advocate, and sound and music nerd. Currently leading product design at Clutch. Bringing advanced AI stats and highlights tor racket sports.