Microsoft Kinect Learns Sign Language

Foreign languages can be tough to pick up on, but sign language may take the cake for being the most foreign of all. With different motions for letters and words and phrases, Microsoft Research Asia and the Chinese Academy of Sciences’ Institute of Computing Technology are coming together to teach Kinect the tricky signs to eventually help hearing-impaired users play.

Starting as a research project, the Kinect is learning translation mode and communication mode. This means that the program can translate hand movements into words and sentences when in translation mode, and when it switches over to communication mode it displays a virtual human that motions sign language back for deaf users.

The magic behind the mystery lies within the 3D trajectory matching where the program digests each of the hand movements and matches them to a word between the probe word and gallery. First, when a user signs a word in translation mode, the Kinect uses 3D trajectory by streaming the visual and depth of where the user is standing and their motion. Next, normal trajectory is used to resample the linear movement and the Kinect digs through its gallery trajectories. Resampling compares the motion to the same sign at different speeds and lengths and exaggerations of the sign. Then, the recognition is based on how it matched to a word in the gallery.

This algorithm makes it possible for the hearing-impaired to play the game that’s sweeping the country. So far, the research team has a database holding 239 sign language words in Chinese. Each word is re-recorded five times to ensure that the signed motion is read and picked up by the system. One is set as the main probe and the others are stored in the system’s gallery.

Project Manager, Guobin Wu, is in charge of the sign language project at Microsoft Research Asia and believes that this program can help improve information technology and should be used to improve people’s lives.

This technology has been used in the past using glove data and web cameras, and even with a large vocabulary, the accuracy was not where it should be. That’s where Microsoft found the loophole and is trying to fill it. The research paper from Microsoft states, “Based on our proposed 3D trajectory matching algorithm, a sign language recognition and translation system is built to connect hearing impaired community and normal persons.” When the Kinect research is complete and the application starts moving toward a consumer release it is expected to be better than any other hearing-impaired system available. The abilities of Kinect to date include detecting depression, acting as a virtual dressing room, and help in military training.