Self-levitating displays allow for physical interaction with mid-air virtual objects

In what’s being billed as a breakthrough in programmable matter, a team of researchers at Queen’s University has released details on a project in which they’ve seemingly bridged the gap between the physical world and virtual reality.

The technology, referred to as BitDrones, could see applications in areas such as 3D modeling, gaming, molecular modeling, medical imaging, robotics, and online information visualization. 

“BitDrones brings flying programmable matter, such as featured in the futuristic Disney movie Big Hero 6, closer to reality,” says Dr. Roel Vertegaal, a University professor who led a team of students in developing this technology. “It is a first step towards allowing people to interact with virtual 3D objects as real physical objects.”

The team used nano quadcopters to create three types of BitDrones. The purpose in doing this was to have each one represent displays of different resolutions. “PixelDrones” – pictured above – have one LED and a small dot matrix display. “ShapeDrones” have a light-weight mesh and a 3D printed geometric frame to serve as building blocks for 3D models. 

“DisplayDrones” are outfitted with a curved flexible high resolution touchscreen, a front-facing video camera, and an Android smartphone on board. All three have reflective markers on them to allow the team to individually track and position them in real time using motion capture technology. The system supporting these drones is also designed to track the user’s hand motion and touch which, in turn, allows for the physical manipulation of voxels in space.

“We call this a Real Reality interface rather than a Virtual Reality interface,” says Dr. Vertegaal. “This is what distinguishes it from technologies such as Microsoft HoloLens and the Oculus Rift: you can actually touch these pixels, and see them without a headset.” 

During demonstrations, the technology is used to physically explore a file folder by touching the folder’s associated PixelDrone. As the folder opens up, its contents are shown by other PixelDrones flying in a horizontal wheel below it. Files in this wheel are then browsed by the user by physically swiping drones to the left or right. 

In terms of the ShapeDrone demo, the team was able to manipulate the drones representing building blocks, and create 3D models in real time. 

And when it comes to the DisplayDrones, the extra devices included on this particular BitDrone system gives the user a telepresence around the drones; that is, he / she can move around locally through a DisplayDrone set-up thanks to Skype software. The system uses the program to track and replicate all of the user’s head movement, and the drones replicate it on the other end, so as to allow the user to virtually inspect a location from a remote location. 

Currently, the system only supports a dozen of 2.5” to 5” sized drones, though the team is actively working to scale this up a bit more — the goal is to have it capable of supporting thousands at a time. Furthermore, they want the drones to measure a bit smaller: no more than a half-inch in size, so that users can render more seamless, high resolution programmable matter. 

Video of the BitDrones demonstration is below:

Via Queen’s University