VR advancement renders a matching virtual ball in motion, its predicted trajectory, and a target catching point lying on the predicted trajectory.

By Brian Santo, Contributing Writer

Researchers at
Disney have devised a virtual reality (VR) system that lets a user play catch
with another person using a real ball. But this is not the more well-known
augmented reality (AR). The ability to detect a real-world object while still
immersed in a virtual world is sort of the opposite of AR. Electronic Products
is dubbing this subcategory of VR “augmented virtuality” (AV). 



Augmented reality
(AR) provides a virtual overlay on reality, usually using transparent glasses.
Imagine attending a sports stadium and being able to watch the game, yet still
get stats or video replays presented in a corner of your vision. Alternatively,
imagine how much easier it might be to put together an Ikea Brimnes daybed if
you can refer to an animated overlay that matches the parts in your hand
exactly, showing you precisely how they fit together.

The reverse is also
possible: overlaying a real-world image onto the virtual world. Creating
digital images of real world objects and inserting them in a virtual world in
real time has been accomplished before. Intel, for example, has an AV component
in its VR technology development program, which it calls Project Alloy.

Intel, for example, has an AV component in its VR technology
development program, which it calls Project
. At the company’s annual
developers conference last summer, CEO Brian Kzranich participated in an
on-stage demonstration
in which an Intel employee was immersed in a VR world; the view from the VR
headset was doubled on a flat screen above the stage. The employee, blind to
the real world, wandered perilously close to Kzranich, at which point the VR
system imported real-time video of Krzanich (frantically making warding-off
motions) into the VR world. What looked like an inevitable collision was

That was an impressive demonstration, but Disney has
significantly upped the ante on it with its ball-catching demo.

“It takes acute visual sensing, prediction, temporal
planning, and refined motor skills…,” the researchers write in their
describing the system. And that’s just for the human doing the catching. The VR system
has to take into such account such factors as “system latency, rendering of
depth cues, frame rate, tracking precision, and registration” which affect

What the
researchers created was a combination tracking and rendering system that
allowed a user in the VR space to see a virtual ball that accurately portrayed
a real ball’s motion. The user could also see both a projected trajectory of
where that ball was going as well as a suggested target point for placing their
hand to catch that ball. With a VR representation of their real hand’s movement
added to the mix, the system essentially showed the user how to catch the real
ball that was coming at them while remaining in the virtual space.

Disney’s video tracking system captures video at 120 frames
per second (fps), almost five times the 24 fps used to film most motion
pictures. The display is an Oculus CV1, using the Unity 3D game engine on a
Windows 10×64 system based on an Intel Xeon chip backed by an Nvidia GeForce
graphics processor. The researchers highlight that they are using a relatively
new mathematical approach, called Unscented Kalman Filtering
(developed around the late 1990s) to perform predictive estimates of the ball’s

Even with all that, the researchers had to scale back the
rendered environment to the absolute basics (wall, floor, baseball, flat paddles
in place of hands) in order to minimize both the rendering frame rate (140-150
fps) and system latency – and the Disney crew reported there were still latency

Disney combined three visualizations, including rendering a
matching virtual ball, rendering the predicted trajectory
of the ball, and rendering a target catching point lying
on the predicted trajectory. This video from Disney shows
the differences among the three, which include changes in how the catcher
catches the ball (a result the researchers apparently did not anticipate).

As extraordinary as Disney’s ball-catching demonstration is,
it’s still a rudimentary system that can and will be refined. The Disney
experiments were with underhand tosses. No one would be insane enough to put on
a headset and try to catch an Aroldis Chapman fastball.

Given time, though, that might become possible.