MIT creates technique to “touch” objects in videos

Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed an imaging technique called Interactive Dynamic Video (IDV) that allows you to “touch” objects in video. With tempting recipe videos and funny animal clips regularly flooding the internet, MIT may have just provided tech that fulfills everyone’s secret wish of video interaction. Using traditional cameras and algorithms, IDV software studies an object’s almost-invisible vibrations to create video simulations that users can virtually interact with.

“This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space,” said CSAIL PhD student Abe Davis. “By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos.”

While the reality of IDV isn’t exactly aligned with interactive concepts such as Smell-O-Vision, it would allow for testing structural designs in videos and reduce the need for CGI green-screens in filmmaking. The notion is better explained through Davis’s example featuring Pokémon Go. Although the app inserts virtual characters into the real world, IDV can enable virtual objects to interact with environments in specific ways. If a Pokémon character were to bounce out of a bush, an IDV user could command the character to interact with the bush even more.  

To simulate objects, MIT researchers examined video clips to track “vibration models” at different frequencies that represent all of the ways an object can move. Once they’ve identified those kinds of shapes, researchers can then predict how the objects would move in new situations. Crafting 3-D models is an expensive process, but Davis’s experiments have proven that even short video footage can be enough to create realistic simulations. Some of his current IDV functions include pushing and pulling an image and even interacting with some existing YouTube videos.

“If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond[ing] to existing forces and assume that it will respond in a consistent way to new ones,” said Davis.

With the possibility of cutting down the cost of virtual reality (VR) development, IDV could also prove to be extremely helpful in the growing wave of VR and augmented reality (AR) technology. “When you look at VR companies like Oculus, they are often simulating virtual objects in real spaces,” said Davis. “This sort of work turns that on its head, allowing us to see how far we can go in terms of capturing and manipulating real objects in virtual space.”

For a closer look at this new technology, check out the MIT research video below.

Source: EurekAlert!, TechCrunch