Icelandic singer and actress Björk made history last week by conducting the world’s first live motion capture press conference at the European launch of the Björk Digital exhibit—an interactive VR experience that spans her career. After experiencing the exhibit, press attendees engaged with Björk during the Q&A session through a virtual avatar, which captured her movements and reactions in real-time via the Unity 3D engine. The artist could hear, see and respond to the audience (located at London’s Somerset House) from Reykjavik, Iceland.
Björk has been a long-time supporter of immersive technologies such as virtual reality, which has become most evident with her 2015 album, Vulnicura. Four songs from the album have been transformed into 360-degree VR experiences, with plans to convert the remaining five songs as Björk Digital tours the world. Meanwhile, the live motion captured press conference represents a major milestone in technology, showing the potential VR applications for other live performances and entertainment experiences such as concerts, sports events and speeches.
Marcos Sanchez, head of global communications at Unity Technologies and Andrew Melchior, the technical producer of Björk Digital from Third Space Agency, spoke to [a]listdaily about how they brought the artist’s vision to life.
How did Unity become involved with Björk’s digital exhibit and real-time motion captured press conference?
Sanchez: We were approached by the team that put together the press conference after they had chosen Unity to be the platform on which they would base the experiences. We love working with artists who are pushing the boundaries between technology, art and people. They really did all the heavy lifting; we just worked in an advisory capacity.
Can you describe the Björk digital exhibit?
Melchior: The Björk Digital exhibit is an eclectic mixture of virtual reality, immersive audiovisual technologies, real instruments, a 5.1 cinema room and a tablet based ‘Biophilia’ music making application.
The foundation of the touring show is the brilliant ‘Black Lake’ immersive film installation created by Björk with James Merry and director Andrew Huang for MoMA in New York. This features two large projection screens showing versions of Andrew’s amazing piece shot in the subterranean lava caves of Iceland.
Then we have three mobile 360 film virtual reality experiences for the songs ‘Stonemilker’, ‘Mouth Mantra’ and ‘Quicksand’. These extraordinary audiovisual works by Björk were done in collaboration with her creative collaborator James Merry, Andrew Huang, Jesse Kanda, Dentsu Lab in Tokyo, Warren du Preez and Nick Thornton Jones. They transport the viewers to visit Björk on a beach, look out of her actual mouth and then fire augmented reality visuals in and out of her amazing ‘Rottlace’ face mask in linear 360 videos.
For this section of the Björk Digital exhibition, we use mobile VR players to let visitors watch the 360 VR films on a phone with the best quality headphones to make sure the dynamic, immersive audio is as great as the visuals. The 4K equirectangular 360 films are hung around a dome created in a special Unity shell to make robust standalone applications that are suited to the play, pause, rewind and repeat of the gallery environment and are capable of utilizing the full audio rendering capabilities within the Unity authoring platform.
The last piece, ‘Notget’ is a real departure creatively and technologically from the linear 360 films and uses the virtual reality real-time game engine in Unity, a fully featured high-end head mounted display, fast computers with powerful video graphics cards and room tracking cameras. This places the visitor into a mysterious, and dynamic computer generated environment featuring Björk in an amazing 3D motion capture performance directed by the visionary directors Nick Thornton Jones and Warren du Preez.
How did Björk explain her vision and what she wanted to do?
Melchior: I find the creative and technical process with Björk, who has such breadth and depth as an artist, is always an iterative and innovative journey with her and her creative, production and management teams.
Björk has always absorbed and experimented with a wide spectrum of horizon scanning technical innovations, VR being the latest in a long line of technologies she has implemented creatively in her recording career. In this instance, we first all met while discussing augmented reality (AR) and how real-time holographic media was starting to become a real possibility with emerging devices and opportunities for new layers of immersion and interaction. Devices being ushered in, such as those being presented by Google’s mixed reality startup Magic Leap and their counterparts from Microsoft and Apple, gave us all some exciting pause for thought about the adjacent possible.
Björk and James were excited about the creative potential for the real world scans we made of Iceland with the X-Rez team and their LIDAR (light and radar) drone captures of the countryside during the making of Black Lake. Björk, James and Andy Huang had a rich palette of 3D objects and assets from the post production outputs from the ‘Black Lake’ film, and they were able to insert these materials into an object-based world browser concept as a kind of VR collage.
The full downloadable Vulnicura VR album experience has been designed by them to test the musical boundaries of this new medium and place the viewer in the middle of an Icelandic volcanic tundra, using the frozen lava motif of Björk from the ‘Family’ moving album cover as its central navigation and song browser. Above this striking frozen lava sculpture of Björk are visual ‘totems’ that depict the album song titles. The user then navigates around each experience by staring at the totem of their choice and teleporting into the distinct pieces.
The overall effect and impact of this first VR Album plays very much to Björk’s passionate interest in Icelandic ecology and uses VR to allow the viewer to connect with the musical landscape that offered Björk her inspiration for the tracks on Vulnicura.
How was Björk’s avatar developed?
Melchior: The amazing streaming avatar was designed by Andy Huang for Björk using scans of her body and face augmented with his own masterly digital paintbrush skills featured in the soon to be premiered ‘Family’ VR narrative. It is based on sketches Björk made and three years of discussions between her, Andrew Huang and James Merry.
The model created by Andy was given to our super skilled Unity developers and animation team Twisted Oak, in San Francisco, who rigged the model with the experienced assistance of Unity Technologies engineers and The Imaginarium Studios motion capture specialist teams based in Ealing, here in the UK. The team in Iceland used MotionBuilder to capture and stream Björk’s motion capture performance data from the Icelandic Multimedia School in Reykjavik live into Unity, which was running in real-time at Somerset House in London and displayed for journalists on the Björk Cinema room screen with 5.1 audio.
What does the virtual interactive press conference represent for the future of VR?
Sanchez: We believe that some of the most compelling applications for VR will have a social component and will involve transcending traditional physical boundaries so people can interact in other realities. At some point in the near future, that same press conference could be done completely virtually with each person putting on a headset to enter a room as the avatar of their choice—talking and interacting just as they would in a traditional press conference. Imagine the holodeck, and you see where we’re going.
How do you think other brands take advantage of real-time motion capture?
Sanchez: Motion capture can be used in everything from gameplay to movies, to virtual experiences. By capturing people’s real physical movements and inserting into other worlds, you’re offering them the ultimate in immersive experiences, which gives brands the emotional connection that they seek. I don’t think we really know exactly what those will look like yet—it’s a brave new world, and were just in the experimentation stages when it comes to understanding what works and what doesn’t. That said, we’ve already seen some amazing things including the fully immersive Ghostbusters Experience at Madame Tussauds that was created by The Void. Expect to see more as people get even more creative.
