IBM's Watson to compete on Jeopardy

[coopasonic]

It's a product demo for Watson. It isn't supposed to be a fair fight.

[Jeff V]

It's a product demo for Watson. It isn't supposed to be a fair fight.

And like most demos of any sort, they are contrived and invariably useless. I think I was more impressed when they simply strove to be better at chess.

[The Meal]

They showed a mechanical system for how Watson buzzes in on Monday's show. I wouldn't call that a contrivance.

What is contrived is that Watson gets the written text of the question inputed directly (he doesn't have to parse Trebek's audio reading of the clue), though this is only a slight contrivance, as the players also are reading the same text off the monitors in front of them (in fact, my understanding is that any successful Jeopardy! player reads the clues instead of paying attention to the reading of the clues as it gives them more time to process).

I think calling this a contrivance in Watson's favor is intentionally inflammatory speech with the intent of riling up forum users with some jackass argument claiming the ends-justify-the-means in terms of generating an interesting debate. I won't be goaded like that and won't be addressing any additional posts in this thread which I perceive are generated in this spirit. I much prefer to engage in an honest discussion with like-minded individuals rather than participate in someone else's psychology experiment.

[mike]

Q.
Buzz-in strategy
On last night's show, I noticed you buzzing in even when you didn't know the answer right away, taking a second after Alex called on you to finish reading the question and give an answer. In your opinion, is this the only way to beat Watson?

A.
Ken Jennings :

Good human players do this all the time: you buzz when you see something that trips some "This looks familiar!" switch in your brain and count on dredging it out in the five seconds after Alex calls on you.

Watson can't do this: it only buzzes once it has an answer in mind and a sufficiently high confidence interval. As weird as it sounds, yes, the human brain still has a speed advantage over a 2,880-processor-core computer.

http://live.washingtonpost.com/jeopardy ... nings.html

[Freezer-TPF-]

I was surprised by Watson's epic fail in Final Jeopardy. I guess it did not parse the category correctly. The array of question marks in its answer were hilarious, as was its somewhat strange betting figures for the Daily Doubles and Final.

I think the humans' only chance is to beat Watson to the buzzer by anticipating the exact moment when Alex finishes reading and the buzzers unlock. Max leveraging for all Daily Doubles and Final Jeopardy would also be necessary. Maybe Watson will get a screwy category that will generate wrong answers with high confidence, but it was pretty strong during Double Jeopardy last night.

[Jeff V]

Q.
Buzz-in strategy
On last night's show, I noticed you buzzing in even when you didn't know the answer right away, taking a second after Alex called on you to finish reading the question and give an answer. In your opinion, is this the only way to beat Watson?

A.
Ken Jennings :

Good human players do this all the time: you buzz when you see something that trips some "This looks familiar!" switch in your brain and count on dredging it out in the five seconds after Alex calls on you.

Watson can't do this: it only buzzes once it has an answer in mind and a sufficiently high confidence interval. As weird as it sounds, yes, the human brain still has a speed advantage over a 2,880-processor-core computer.

http://live.washingtonpost.com/jeopardy ... nings.html

Poor Jennings must have become retarded since his glory days, then. Was there even a single instance where Watson knew the answer but was beat at the buzzer by one of the human players? It didn't seem to be the case. From what I understand, the buzzers aren't activated until Alex stops talking, so while they can start clicking away at any time after the clue is displayed, only the first one that registers when the buzzers are activated counts. Watson, it seems, has an unmistakable advantage here.

Most computer games design AI opponents to operate within the same constraints as a human. How much fun would it be to play a game where with the AI always shoots first, and always delivers a lethal kill shot? This is, in effect, what Watson is doing.

And Meal, my point is this isn't a fair challenge, it's a contrived product demo. Jennings and Rutter are serving the same functional purpose as the models at the Auto Show currently in town -- a useless "demo" of the products since you can't actually drive the cars there. If they dispense with the pretense and just have the computer work its way through a Jeopard board, it would be as entertaining and probably a little more informative without the distractions of having Jennings and Rutter along side of it.

This is a far cry IMO from Big Blue vs. grand masters of chess.

After getting yet another bewildering transcription from Google Voice, I think a more impressive demo would be to see how well it can actually convert speech to text. That would at least have some useful application. Since Google is one of the few companies that could make use of a Watson, it might even drum up some business for IBM.

[Defiant]

The question is, is it an unfair advantage, or simply something Watson is good at?

Watson does have a big advantage in this regard, since it can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard. But I wouldn't call this unfair ... precise timing just happens to be one thing computers are better at than we humans. It's not like I think Watson should try buzzing in more erratically just to give homo sapiens a chance.

http://www.thenewstribune.com/2011/02/1 ... ntage.html" target="_blank

What is contrived is that Watson gets the written text of the question inputed directly (he doesn't have to parse Trebek's audio reading of the clue), though this is only a slight contrivance, as the players also are reading the same text off the monitors in front of them (in fact, my understanding is that any successful Jeopardy! player reads the clues instead of paying attention to the reading of the clues as it gives them more time to process).

I wouldn't call it a slight contrivance. I would assume it's a lot quicker to directly interpret a string of maybe a couple of hundred characters than to process that string through our eyes from a distance.

And then theres the advantage that Watson has a ton of information thrown at it, that it can access with complete recall. Human memory doesn't work that way. But is that an unfair advantage or simply something it's better at?

[Kasey Chang]

I disagree about the "contrived". Jeopardy is a very unique challenge as it involve language analysis and machine learning.

Jeopardy use a lot of puns, rhymes, and such. If you watched the NOVA special, you'll see that when Watson first got started, he was simply using the terms in the question, goes through his database looking for commonalities, and rate the various answers. It's not working. His correct answer rate is worse than even an average IBM employee.

Initially Watson don't even get the wrong answers inputted, so he would give the same wrong answer as someone else. The correct answer later was inputted so Watson can update his sorting algorithm on the fly. If his "train of thought" was wrong, the 'right' answers would give him a better idea what the category was looking for.

Chess play is completely different. Computer plays chess by basically doing "calculating ahead". It simply calculates bazillion moves ahead by predicting what you can do to the board. What if you do this? What if you do that? What should I move in response to this? To that? It calculates ahead by as much as possible. Human brains don't do that, not even grandmasters. What grandmasters do is they have certain patterns in their head. They see a pattern, and they instantly remember what needs to be done to counter the pattern. This was covered in one of those PBS specials. They got one of the female masters looking at a chess board. That board was from a real game. She looked at it for only a few seconds. And she can recreate it on a chess board. However, if she's asked to look at a board with randomly placed pieces, it did NOT come from real chess moves, so she can't remember them.

Very interesting, isn't it? Computer playing chess is basically exploiting the speed advantage they have, while human playing chess is exploiting the pattern recognition engine in our brain. It's not exactly a fair comparison.

Watson is closer to how humans playing Jeopardy! as we recognize the terms in the statement / question, recognize the context of the terms, and actually "understand" the question as it as intended. If one asks "Are you in the Matrix?" The "Matrix" clearly refers to the movie, not the mathematical representation of an array of numbers. Yet a computer would have no idea about that unless it knows about the movie and understands "in the Matrix" reference.

yes, it's a tech demo. No, it's not as contrived as you make it out to be.

[Defiant]

Jeopardy is a very unique challenge as it involve language analysis and machine learning.

While I don't consider the game exactly fair, I do agree that this was an impressive accomplishment by IBM.

I did find it interesting that Watson was doing better later in the game (with harder questions). But I wonder if this was just because he was doing well in comparison to the other two doing less well.

[Jeff V]

yes, it's a tech demo. No, it's not as contrived as you make it out to be.

<sigh> It is contrived precisely because it IS a tech demo and not a competition as it is advertised to be!

[stessier]

And then theres the advantage that Watson has a ton of information thrown at it, that it can access with complete recall. Human memory doesn't work that way. But is that an unfair advantage or simply something it's better at?

Simply something it's better at.

[Kasey Chang]

yes, it's a tech demo. No, it's not as contrived as you make it out to be.

<sigh> It is contrived precisely because it IS a tech demo and not a competition as it is advertised to be!

Nope, because Watson actually have to audition for the game just like any other Jeopardy contestant!

[Jeff V]

So, this $1 million challenge was open to everyone, yet it just happened to draw the two most successful humans and a computer designed with this in mind? Pardon my skepticism.

[tgb]

I agree with Jeff - this competition was designed specifically with Watson in mind - no audition was necessary (other than the techs reporting that Watson was "ready").

[Kasey Chang]

I agree with Jeff - this competition was designed specifically with Watson in mind - no audition was necessary (other than the techs reporting that Watson was "ready").

You need to watch the NOVA special.

[mike]

Nah, Jeff is right. Watson has pointy elbows, and isn't near as sexy as MS BOB.

[mike]

Can someone start a new thread concerning how important this is, without the retards involved?

[tgb]

Can someone start a new thread concerning how important this is, without the retards involved?

Classy.

[Defiant]

And then theres the advantage that Watson has a ton of information thrown at it, that it can access with complete recall. Human memory doesn't work that way. But is that an unfair advantage or simply something it's better at?

Simply something it's better at.

While it probably wouldn't have helped (given time issues) would it be "fair" to allow Human players to access a (not-networked) laptop?

Speaking of not being networked, I wonder how big the entire text portion of the internet would be, and whether Watson could just have an offline copy of it. Hmm

[Quaro]

Watson's mechanical buzzer can be beat two ways. On occasion a human can buzz within 10 to 15 ms and beat it straight out by anticipating the timing correctly. The other way is that sometimes Watson takes longer to process the question than Alex takes to read it, and it doesn't buzz in until it has arrived at the answer. It's not one nanosecond, it's any where from a couple to 7 or 8 seconds, which can be too long.

I do think there's a bit of a argument that there is some contrivance here. They could have spent time optimizing the mechanical buzzer to win an even higher percentage. And for the other case, they could have worked on a buzzing estimator that always finishes before Alex is done speaking and decides whether to buzz, and then stalls, using the extra five seconds as additional processing to arrive at the answer. Both of those would probably make Watson dominate even more, but more importantly, don't address the core challenge.

Jeopardy is a game of reflexes more than anything else, when between humans. Against a computer the competition has to be different.

The point of Watson not being connected to the Internet isn't because Watson might do a Google search -- Watson has whatever raw data is needs internally, I'm sure. The hard part is knowing what to search for and how to interpret the results. But by not being on the network, the machine stands alone, and isn't receiving secret help.

[Defiant]

Jeopardy is a game of reflexes more than anything else, when between humans.

I thought it was a game of trivia and word play. Certainly thats whats supposed to be the big accomplishment with Watson.

IMO, the best way to make the game "fair", though it would have changed the rules of the game, would have been to allow all the players to independently give an answer within a set amount of time, and randomly choose the ordering to consider player answers. This would have taken the physical nature of answering the question and make it that of solving the word play.

[tgb]

Highest ratings in 6 years.

I'm guessing the highpoint 6 years ago was during Jennings' run.

[Jeff V]

Jeopardy is a game of reflexes more than anything else, when between humans.

I thought it was a game of trivia and word play. Certainly thats whats supposed to be the big accomplishment with Watson.

IMO, the best way to make the game "fair", though it would have changed the rules of the game, would have been to allow all the players to independently give an answer within a set amount of time, and randomly choose the ordering to consider player answers. This would have taken the physical nature of answering the question and make it that of solving the word play.

That shouldn't be necessary. Any modern game designer worth anything could program a variable time delay that would match the ability of the human players.

It's not as if the human players were baffled by the toughest board in the history of the game. A fair game should have roughly given equal distribution to all three players -- that's all I'm saying. If the goal was to provide a fair competition, this clearly isn't happening (and with regards to an earlier comment about 7-8 second decision times, I think the question times out after three or five seconds).

Jeopardy is buying a ratings bonanza by putting on the game show equivalent of a WWF title match. Unless the machine includes a Cliff Claven routine, Rutter and Jennings have no chance -- had no chance. The amused look on their faces though suggests it's all going according plan. As a tech demo, it's an amusing, if pandering, display. As a competition though, I'd rather watch a Big Blue chess match.

[coopasonic]

It is a fair competition. Both the human players and Watson get the question at the same time and have the opportunity to buzz in at the same time. In fact from that perspective, it's unfair in that the humans can buzz in without having an answer where Watson has been restricted to wait until it has an answer to buzz in.

Now the fact that Watson is simply much better at clicking the buzzer and has terabytes of memory full of encyclopedic knowledge doesn't make it unfair it just makes it very uneven.

It's like having the winners of the Little League World Series play the Yankees. As long as both teams are heldto the same rules then it is fair. It's not really very sporting, but it's fair.

[LordMortis]

The game is a game of speed and accuracy of understanding. To make the game "fair" all they needed to not have the questions "scanned in" but rather had Watson "hear" or "read" the question exactly as the other players do. If Watson is faster to the buzzer then that is definitely part of the game. Handicapping Watson for the game is what makes the game for fanfare.

There should be no leveling of the playing field, then what makes the game interesting is simply Watson's performance and seeing where it fumbles in real time.

[LordMortis]

It is a fair competition. Both the human players and Watson get the question at the same time and have the opportunity to buzz in at the same time. In fact from that perspective, it's unfair in that the humans can buzz in without having an answer where Watson has been restricted to wait until it has an answer to buzz in.

Now the fact that Watson is simply much better at clicking the buzzer and has terabytes of memory full of encyclopedic knowledge doesn't make it unfair it just makes it very uneven.

It's like having the winners of the Little League World Series play the Yankees. As long as both teams are heldto the same rules then it is fair. It's not really very sporting, but it's fair.

Agreed. If Watson thinks it's got the answer and needs a moment to "think" about it, it should be able to to buzz in and get that three second time out or whatever it is they do on Jeopardy.

[coopasonic]

The game is a game of speed and accuracy of understanding. To make the game "fair" all they needed to not have the questions "scanned in" but rather had Watson "hear" or "read" the question exactly as the other players do. If Watson is faster to the buzzer then that is definitely part of the game. Handicapping Watson for the game is what makes the game for fanfare.

There should be no leveling of the playing field, then what makes the game interesting is simply Watson's performance and seeing where it fumbles in real time.

They could have added a camera and OCR routines but that's not what Watson is about. I doubt it would have made much difference in timing. OCR is simple given Watson's resources.

[Jeff V]

It is a fair competition. Both the human players and Watson get the question at the same time and have the opportunity to buzz in at the same time. In fact from that perspective, it's unfair in that the humans can buzz in without having an answer where Watson has been restricted to wait until it has an answer to buzz in.

Now the fact that Watson is simply much better at clicking the buzzer and has terabytes of memory full of encyclopedic knowledge doesn't make it unfair it just makes it very uneven.

It's like having the winners of the Little League World Series play the Yankees. As long as both teams are heldto the same rules then it is fair. It's not really very sporting, but it's fair.

Agreed. If Watson thinks it's got the answer and needs a moment to "think" about it, it should be able to to buzz in and get that three second time out or whatever it is they do on Jeopardy.

The thing is, it doesn't, it has the decision already by the time it buzzes in, which it can do the instant the buzzer is active. Perhaps if it was working a mechanical device like the players and had to anticipate the variable moment the buzzer would be active, then it could work. But last night the machine seemed to win 100% of the buzz-ins when it was reasonably confident of an answer. Given the caliber of the human opponents, this is a highly unlikely outcome if all is fair and equal.

[coopasonic]

Perhaps if it was working a mechanical device like the players and had to anticipate the variable moment the buzzer would be active, then it could work.

Watson uses the exact same buzzer as the humans.

http://www.wired.com/epicenter/2011/02/ ... son-speed/" target="_blank

W: How does Watson actually ring its buzzer?

B: Watson has a mechanical button-presser. It uses the same signaling device [the button] that the human competitors use in the game. Once Watson has decided that it wants to ring in because it has found an answer with a high-enough confidence, and it receives the signal that the buzzers are open and you can ring in, it then has to trigger the mechanical button presser and mechanically press the button.

W: At the beginning, Watson didn’t have a mechanical button-presser, did it?

B: Early on, during some of early sparring games [Watson played approximately 135 sparring games], the initial implementation did not have a mechanical button presser, and in fact Watson actually sent an electronic signal back to the Jeopardy control system indicating that it wanted to ring in. Ultimately Jeopardy and IBM decided that it would be more fair if we used the same signaling device as the human players, and so we then added the mechanical button-presser. We used that for all of our 55 matches against the Tournament of Champions players last fall.

W: So how do you respond to critics who say Watson has an unfair advantage because it can ring in faster?

B: Ultimately, this is being portrayed as a human vs. computer competition and there are some things that computers are going to be better at than humans and vice-versa. Humans are much better at understanding natural language; computers are better at responding to signals.

[Odin]

[YellowKing]

Just talking to people at work, reading this thread, etc. I'm a bit puzzled at some of the reactions. Some people think it's cool (including me). Others are pissed because they feel like Watson has an unfair advantage, as if a Jeopardy game is the final arbiter of whether robots or humans will rule the earth. Still others are mad that Watson isn't *more* intelligent.

In my mind, it's just a really neat proof of concept. I don't think anybody is claiming that this is true human equivalent AI. And I don't think anybody is claiming that every aspect of the match is 100% fair. It's something to spark the imagination. It's one of those baby steps that is taken early in the history of every great invention. If nobody takes that step and tries it, we'll never really know what the potential is. In that regard I think it's an important milestone in the history of artificial intelligence.

[Kasey Chang]

The point is this is the MAJOR step on the way to truly recognize speech.

Right now speech recognition is purely sound pattern recognition, with no understanding of the words behind them. With Watson's technology, it will understand context and auto-correct errors or guess meaning based on context, thus vastly increasing accuracy.

When I say "go to the store", you would have no idea which store I was referring to unless you know me, and otherwise understand the context I was referring to, i.e. the corner store around the block.

Location-based services are one of the steps on the way to understanding.

[Odin]

There's a short article on Gizmodo. I've quoted the bulk of it below:

Watson Crashed Multiple Times on Jeopardy, Plus Other Watson Tidbits

Casey Chan — Watson Crashed Multiple Times on Jeopardy, Plus Other Watson TidbitsIf you saw the second night of Jeopardy's Watson vs Man and Watson's flat out dominance, I wouldn't blame you for being a little bit sad about humanity. Thankfully it's not straight to SkyNet for Watson, turns out the machine is a little unreliable and prone to crashes.

PBS was filming a documentary on Watson and was around for the taping of the show. To quote Michael Bicks, one of the PBS producers, "[Watson] crashed a bunch of times it took over four hours to tape the show—most of the delays were due to crashes". Four hours to tape a half hour show is a pretty long time, especially considering that normal Jeopardy shows can get done in under 2 hours.

Ken Jennings was also at the taping of the show (if you didn't notice him) and [seemed] to be frustrated at Watson's excellence at buzzing and finally knows what it feels like to go against a 'Ken Jennings':

"Watson does have a big advantage in this regard, since it can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard."

The team that built Watson actually pattered the machine off of Jennings himself:

"The Watson team told me two things after the match: that the idea for Watson was born after watching my 2004 streak on Jeopardy, and that they watched LOTS of tape of me while honing its skills. "There's a lot of you in Watson," one guy said."

Man spawns machine. Machine beats man. A story as old as time.

[Defiant]

PBS was filming a documentary on Watson and was around for the taping of the show. To quote Michael Bicks, one of the PBS producers, "[Watson] crashed a bunch of times it took over four hours to tape the show—most of the delays were due to crashes".

I wonder how much leeway Jeopardy would give to a contestant who took naps during the show.

[mike]

Despite reports, Watson did not crash during 'Jeopardy' taping

Read more: http://news.cnet.com/8301-17852_3-20032 ... z1EBBlM3tw" target="_blank

[tgb]

Kudos to Ken Jennings for having a sense of humor.

[Koz]

The point is this is the MAJOR step on the way to truly recognize speech.

Right now speech recognition is purely sound pattern recognition, with no understanding of the words behind them. With Watson's technology, it will understand context and auto-correct errors or guess meaning based on context, thus vastly increasing accuracy.

When I say "go to the store", you would have no idea which store I was referring to unless you know me, and otherwise understand the context I was referring to, i.e. the corner store around the block.

Location-based services are one of the steps on the way to understanding.

I'm pretty sure Watson gets fed the text of the questions and doesn't use any kind of speech recognition.

[Quaro]

You could really see Watson's processing time have an effect on the third night. For many short questions, it had the right answer, but didn't buzz first.

[Grifman]

Watson is closer to how humans playing Jeopardy! as we recognize the terms in the statement / question, recognize the context of the terms, and actually "understand" the question as it as intended.

Actually, no, that's not correct. Much of human knowledge, we just know. Or we can get to 2 or 3 possible answers right off the bat and select the one we think is best of the 3. Watson doesn't work that way at all.

Watson, like Deep Blue is using a brute force method. It has the equivalent knowledge of 1,000,000 books. It takes the key terms, looks them all up, checks their associations comes up with a list of possible correct answers, and then has an algorithm for assigning probabilities as to which of it's possible answers is correct.

[Grifman]

The point is this is the MAJOR step on the way to truly recognize speech.

Right now speech recognition is purely sound pattern recognition, with no understanding of the words behind them. With Watson's technology, it will understand context and auto-correct errors or guess meaning based on context, thus vastly increasing accuracy.

I don't follow. Watson doesn't recognize speech, the questions are typed in. Watson's great at answering typed questions but he doesn't recognize speech at all. Those are two separate things in my mind.