Google Search vs Apple Siri: 1600-question test

Google Search vs Apple Siri: 1600-question test

Intriguing study brings to light most clear-cut evidence to date

BY JEFFREY BAUSCH

Before we get into why Google received a “B+” and Apple a “D”, let’s set some things straight from the get-go: yes, we understand that Siri is supposed to be more A.I. than a search app; yes, we understand that Siri is still considered “beta”; and yes, testing the accuracy of a keystroke-based discovery service versus a voice-activated personal assistant app is like, well, testing the accuracy of a keystroke-based discovery service versus a voice-activated personal assistant app. It’s not entirely fair.

We understand all of this, but quite frankly, all of these excuses, if you will, they’re inconsequential: the fact of the matter is that Google Search and Apple’s Siri are the two most popular ways to discover information using smartphones today.

They not only can be compared to one another, they should be compared to one another.

A recent study compares the accuracy of Google search against Apple’s Siri.

Enter Gene Munster of Piper Jaffray. His company is an investment bank and asset management firm that services clients in the U.S. and internationally with product and sector expertise. Mr. Munster is a managing direction and senior research analyst, who specializes in all things internet.

Why does this even matter? Because if there’s one person whose reports you can trust, it’s this guy.

The comparison

Munster wanted to know how the new kid on the “search” block, Apple’s Siri, compared to everyone’s go-to method of finding information, Google. To determine this, he and his staff went about asking an iPhone 1,600 questions: 800 in a quiet room, and 800 while standing outside on the busy, loud streets of Minneapolis.

Here’s what they found:

• Google understands 100% of the questions (since they’re being typed in, this really shouldn’t come as that much of a surprise).

• Google replies accurately 86% of the time.

• Siri comprehends queries with 89% accuracy in a quiet room, 83% in noisy conditions.

• Siri answers accurately 68% of the time in a quiet room, 62% of the time on the street.

Upon reviewing his results, Munster estimates that Siri is about two years behind Google in its learning curve. Despite this seemingly insurmountable hill to climb, he is optimistic about Apple’s ability to close the gap: “With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google from 60% to 48%.”

Breaking Apple’s dependency on Google

At present, Siri gets 60% of its answers from Google. The rest of the results come from the following resources: 20% from Yelp, 14% from WolframAlpha, 4% from Yahoo, and 2% from Wikipedia.

“Breaking down Siri’s reliance further,” Munster explains, “Google provides 100% of navigation results, 61% of information results, 48% of commerce results and 42% of local results. Among other result aggregators, Yelp provided the most local results (51%) and commerce results (51%), while WolframAlpha provided 34% of information results.”

At Apple’s most recent conference, the company went into detail about some of the changes that we can expect to see when its new mobile operating system, iOS 6, rolls out. One of the biggest and most popular changes — which Munster eluded to earlier — will be the dropping of Google Maps in favor of Apple’s in-house maps system.

Additionally, the company plans to integrate content providers Yahoo! Sports, Open Table, Rotten Tomatoes, and Fandango to provide Siri users with more accurate answers regarding sports, restaurants, and entertainment queries.

Some of Siri’s stumbled responses

So yes, Apple has an idea of what it has to do in order to begin to sway the market away from Google, but in looking at some of the responses that Munster was given during his test, the road to achieving this is a long one.

Very, very long as a matter of fact.

Here are some samples:

• What team does Peyton Manning play for?

– Response: Answer to previous query (this was the most common error).

• Where is Elvis buried?

– Response: I can’t answer that for you (it thought the person’s name was Elvis Buried).

• Where am I?

– Response: Pin dropped in the wrong place.

• When did the movie Cinderella come out?

– Response: Movie theater search on Yelp.

• How do I get from Boston to New York?

– Response: I can only give directions from your current location. I can’t give you directions to a place you are not in.

• What spices are in lasagna?

– Response: Yelp search with lasagna on the menu.

• When is the next Haley’s comet?

– Response: You have no meetings matching Haley’s.

• I want to go to Lake Superior.

– Response: Directions to the company Lake Superior X-Ray.

Personally, I would love to see a Siri commercial where Samuel L. Jackson responds to these misguided answers. ■

Story via: money.cnn.com