Machine learning algorithm copies your likeness and creates a 3D puppet solely from Internet photos

University of Washington researchers are developing a machine learning algorithm that captures a person’s mannerism and reconstructs an accurate, controllable 3D digital model after studying images on the Internet.

The technology could serve as the forerunner of what will eventually become 3D interactive images that immortalize family members and historic figures as digital personas, after studying images from family albums or historic items.

The team has proven that with enough mineable data — such as images publicly available through the Google image search — they can recreate an animated model of Tom Hanks capable of delivering lines the actor has never performed; no expensive in-house motion capture necessary.

“One answer to what makes Tom Hanks look like Tom Hanks can be demonstrated with a computer system that imitates what Tom Hanks will do,” said lead author Supasorn Suwajanakorn, a UW graduate student in computer science and engineering.

Achieving this resulted relied on the culmination of multiple technologies, including 3D face reconstruction, tracking, alignment, multi-texture modeling, and puppeteering, all which has been refined over the last five years by a research group led by Ira Kemelmacher-Shlizerman, UW assistant professor of computer science and engineering.

The latest developments, which will be presented in a paper at the International Conference on Computer Vision in Chile this month, include the ability to map facial expressions and the muscular contortions that occur when a person speaks, onto the face of another person. Or in the case of their experiments, the faces of politicians and celebrities.  

By integrating virtual reality, such capabilities can also lend a new level depth to long-distance Skype conversations, allowing participants to “see each other” in three dimensions.

“You might one day be able to put on a pair of augmented reality glasses and there is a 3D model of your mother on the couch,” said senior author Kemelmacher-Shlizerman. “Such technology doesn't exist yet—the display technology is moving forward really fast—but how do you actually re-create your mother in three dimensions?”

Unlike rival technology seeking to fulfill the same goals, the UW computer vision experts wish to recreate a 3D avatar solely from random images.

Contemporary technology seeking to recreate three-dimensional avatars typically require that the subject be scanned with a camera, but the UW computer vision experts wish to accomplish this solely from collections of existing images. The Tom Hanks, Daniel Craig, and George W. Bush replicas were built from 200 images each, that the machine learning algorithm scanned “out in the wild.”

Source: Phys.org