Overview / Usage

Demoed at GDC 2017, OrlandoiX 2016, Austin Game Conference 2016, IDF16, and SIGGRAPH 2016.

Winner of Best of Show at the Intel Game Developer Showcase, Austin Game Conference 2016.

Winner of Most Innovative Use of AR/VR, OrlandoiX 2016.

Shapesong is a concept we are developing that lets people jump into VR and start playing music as easily as banging on a drum. Explore a musical environment where everything you see can be played. Create music loops called memories, or use ones from others, to create an entire band easily by yourself. Discover sounds and interactions to create unique musical pieces that you can share with the world. Our goal with Shapesong is to create a new kind of music experience that blends music, visuals, and performance into one, and to enable and inspire people of all skill levels to create music of their own. Jump in and bring a world to life with music, limited only by your imagination.

Shapesong is being developed specifically for room-scale VR platforms. We are currently prototyping the app and will show more media and information as we progress.

Update 1: SIGGRAPH and IDF build

We've been working on a proof of concept build of Shapesong to demo at both SIGGRAPH and IDF. In this version, we have a forest garden area with some different instruments and interactive visuals for people to explore and play with. There are also a number of sound cartridges that can be used to load new sounds onto instruments. At end of the garden, we have a performance area that brings all of the instruments together so that players can create a song by recording and looping themselves, as well as playing with another character.

With this version we plan to collect feedback from those at the trade shows and use it to define our next build of Shapesong.

Update 2: Austin Game Conference, Teaser Trailer

We just got back from Austin Game Conference, where Shapesong won "Best in Show" at the Intel Game Developer Showcase. Shapesong was well-received, and we can't wait to start sharing new features as they're developed. Our first teaser trailer for Shapesong is up on Youtube (link below) as well, so check it out!

Update 3: GDC 2017

A brand new version of Shapesong was playable for the first time at GDC 2017 this past week, this time in Mixed Reality! Above the GDC show floor at the Intel Sky Bridge, Shapesong was playable in a fully green booth—allowing us to capture the user in mixed reality. Everyone who ran through Shapesong (except for one, sorry Eileen!) was able to walk away with a flash drive containing their experience in Shapesong, as a mixed reality recording. We uploaded a new video of Shapesong (Intel Developer Zone Update for March 2017) and will be uploading some mixed reality videos soon.

Methodology / Approach

We are using VR systems to track users and enable them to interact with virtual music environments. The core components of these systems are VR HMD's and 6DOF tracked controllers, allowing for a fully immersive room-scale tracking volume.

Outside of core user presence, we are also using music frameworks, standards, and technologies. For example, we use MIDI to allow for non VR users to play on our instruments just like they would inside a DAW or other virtual instrument. We also created our own music library to wrap around Unity's basic audio components, allowing us to use music language (tempo, beats, scales, keys, etc) on the settings for our instruments and songs.

Overall our approach has been to create an application that isn't too dissimilar from a DAW so that we can easily transition content from traditional music production environments directly into Shapesong.

Technologies Used

Hardware

• Room scale VR platforms (Rift, Vive)
• Green screen with mixed reality compositing

Software

• Unity 3D
• Photoshop
• Maya
• Substance Painter
• Pro Tools

Intel Technologies

• Alienware Aurora PC
• Intel GPA

Collaborators