You are what your phone thinks you eat, at least when AI suggests recipes based on photos

By Warren Miller, contributing writer

What if you could find out what’s in the food you’re eating simply by taking a picture of it with your smartphone? Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) in cooperation with their colleagues at the Qatar Computing Research Institute (QCRI) are developing a database that may soon allow you to do just that.

While the technology to recognize foods from digital images has been in various stages of development in some years, efforts by researchers have been hampered by a lack of wide-ranging and comprehensive data. To put it simply, there are many different kinds of food in the world, and compiling a database of digital images of all of them is an enormous undertaking. City University in Hong Kong gathered over 110,000 images and 65,000 recipes of Chinese food alone.

The CSAIL research team addressed the problem by using images from popular cooking websites like food.com and All Recipes to create a gigantic database of food dishes and their respective ingredients, creating a much more comprehensive dataset than any that’s come before. They subsequently used that data to create a program that can infer from a simple photograph of a dish what its principal ingredients are, such as eggs, butter, and/or flour. It can even make recommendations for other recipes using similar ingredients.

Your smartphone may soon know exactly what you’re eating, ingredients and all. Image source: Pixabay.

“You can imagine people using this to track their daily nutrition or to photograph their meal at a restaurant and know what’s needed to cook it at home later,” said Christoph Trattner, an assistant professor at MODUL University Vienna in the New Media Technology Department. “The team’s approach works at a similar level to human judgement, which is remarkable.”

Of course, knowing exactly what ingredients were used to prepare your food could dramatically affect your nutritional habits.  No more wondering what’s in that piece of pumpkin cheesecake you just ordered; just snap a quick photo of it with your phone and you’ll have the complete list of ingredients in seconds. You'll know precisely what you are eating, which might alter your choices.

The system developed by CSAIL and QCRI is particularly effective in revealing the composition of desserts, which is fortuitous for those of us afflicted with a sweet tooth. If you ever liked a dish you ordered in a restaurant so much that you wish you could make it at home, the program can tell you exactly what you would need to pick up at the grocery store in order to do just that. Even better, perhaps, by linking with an online ordering app, the ingredients could be delivered directly to your door using a drone delivery service.

One day, the technology may even be able to draw on its recipe database to tell you how exactly how a specific dish was prepared, from how thin to slice the tomatoes in a lasagna to how long to leave a roast in the oven.  Perhaps culinary schools will become a thing of the past, with aspiring chefs learning how to make complex recipes from the comfort of their own kitchens. You may never have a hand steady enough to perfectly sauté mushrooms over an open flame, but you won’t be able to say you don’t know how to.