Bringing Realism To VR

Read any extensive coverage of virtual reality and undoubtedly you’ll come across the word “immersion” at least once or twice. That’s a relatively vague term for the feeling of presence within the virtual world, but it’s as close as we’ll get to providing a singular point of reference for gauging the success of a VR developer in crafting a believable experience.

Most developers utilize similar technique to achieve immersion. They work within the bounds of existing virtual reality headsets and craft small-scale, high-fidelity scenes, with binaural audio. If they do a good enough job, you may find yourself wandering into the uncanny valley. It’s rare to climb the rise on the other side — yet that’s exactly what Awake from virtual reality production house Start VR is attempting to do.

Its roughly 20-minute first episode is a lightly interactive cinematic VR experience that delivers a captivating story around the theme of lucid dreaming. But as intriguing as the narrative is, it’s the volumetric capture technique used to craft it that is Awake‘s most striking element.

The end result is characters with incredibly realistic facial expressions, motions, and gestures. This is no ping-pong-ball motion capture. It’s something else entirely.

SO LONG PING PONG BALLS, HELLO HOLOGRAMS

“The technique that Awake was specifically written around is volumetric video,” explained creator and director Martin Taylor. “So it’s a new type of performance capture that takes the full actor’s performance including all of the wardrobe and the full expression and captures them as a filmic hologram.”

This is very different from the motion capture techniques used to craft most in-game characters for both virtual reality experiences and more traditional 2D games. While those use samples of movements to craft the 3D models that are skinned to bring virtual characters to life, volumetric capture records the entire person. Inside and out, so to speak.

“It’s using a special rig with 106 cameras with depth sensors all pointing towards a cylindrical volume in the center and up to two actors can stand in that volume and deliver their performances,” Taylor told Digital Trends. “All of these cameras and sensors capture that, and lots of processing and software stitches it together into a completely believable and solid digital copy of the performance.”

That gives the developers not only an accurate 3D representation of where those actors are and how they’re posed at any particular time, but a near photo-realistic video impression of them too.

“The raw version of the captures are a huge series of OBJ models which are all uniquely different from one another and then a video texture that’s wrapped over the top,” Taylor said.

Early pioneers in its usage, volumetric capture wasn’t something Start VR could perform alone. It needed Microsoft’s help.