If you forget to tag or add a description when uploading a photo to Facebook, it can be quite annoying to find that picture later on. Well, at least it used to be. The social network just revealed that it built an AI image search system that can “see” things in your photos, even when you forget to add the aforementioned identifiers. According to Facebook, the system uses its Lumos platform to understand the content of photos and videos and to quickly sort through the items you’ve uploaded. 

This means if you can’t remember when a photo was taken but recall the content, you might still be able to quickly locate it. For example, you can search “blue shirt” on Facebook and the system will pull up all of your photos that include a blue shirt, even if they aren’t tagged as such.

For this to work, Lumos has to be able to recognize objects when they appear in photos. Facebook developers used deep learning and a neural network to train the system to identify objects using tens of millions of photos with the proper annotations. Facebook’s fortunate in this respect because its platform already hosts billions of captioned images. Essentially, the model matches search descriptors to features pulled from photos with some degree of probability. 

Basic structure of how object recognition works. Image source: Tech Crunch.

By doing this, the image search is able to pick up on scenes, objects, animals, attractions, places, and clothing. Impressively, the system factors in some degree of diversity so the search results aren’t a cluster of similar images. Though this technology may seem extreme to some, it will help Facebook describe images and video to users who are visually impaired.

Eventually Facebook will apply this technology to its growing video collection. It could be used in the personal context of searching a friend’s video, to find the exact moment her baby son took his first step, for example, or in a commercial context. The latter could help raise the ceiling on Facebook’s potential ad revenue from News Feed.

The race is on

Of course, Facebook isn’t the only one racing to apply recent computer vision advances to existing products. Pinterest has a visual search feature that’s continuously improved upon to let users search images by the objects within them, and last fall, Google open-sourced its own image captioning model that can identify objects and classify actions with over 90% accuracy.

Facebook’s Lumos was built on top of FBLearner Flow, the company’s machine learning platform, and has already been used for over 200 visual models. Aside from image content search, engineers used the system for fighting spam.

Sources: TechCrunch, The Verge